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PREFACE 


This  report  was  prepared  at  SYTRONICS,  Inc.,  Corporate  Headquarters,  4433  Dayton- 
Xenia  Road,  Building  1,  Dayton,  Ohio.  The  work  was  performed  under  Contract  F33615-98-D- 
6000,  Delivery  Orders  (DO)  0001  through  0013.  Inclusive  dates  of  the  research  reported  are 
from  25  September  1998  through  30  September  2005.  Mr.  Ronald  Yates  of  the  Air  Force 
Research  Laboratory,  Human  Effectiveness  Directorate,  Warfighter  Interface  Division 
(AFRL/HEC)  was  the  Program  Manager.  The  purpose  of  this  report  is  to  summarize  the 
achievements  and  successes  accomplished  on  this  program. 
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AFRL  RESEARCH  PROGRAM 


SUMMARY 

SYTRONICS'  tenure  as  support  contractor  to  AFRL  can  be  characterized  as  highly 
effective,  innovative,  and  cost  effective.  Our  company  was  grateful  for  the  opportunity  to  serve 
AFRL  and  looks  forward  to  continuing  to  provide  our  technical  expertise  to  help  the  Air  Force 
Warfighter. 


INTRODUCTION 

SYTRONICS,  Inc.,  was  founded  in  1983  and  has  since  expanded  into  a  multifaceted 
company  offering  broad  expertise  in  three  major  divisions:  Human  Research  and  Systems 
Engineering;  Modeling,  Simulation,  and  Training;  and  Test  System  Development.  Our 
capabilities  include  research  and  development,  applied  engineering,  program  management, 
product  development,  and  product  support  utilizing  a  lean,  highly  productive ,  and  cost-effective 
staff  of  engineers,  human  factors  professionals,  technicians,  training  and  support  specialists,  and 
administrative  personnel  needed  to  achieve  and  surpass  the  requirements  of  our  contracts. 

SYTRONICS  has  extensive  technical  knowledge  in  research  operations  for  advanced 
warfighter  interface  technology  that  enabled  us  to  achieve  AFRL/HEC  laboratory  objectives  in  a 
timely  and  cost-effective  manner.  We  made  many  changes  and  streamlined  operations  over  the 
course  of  the  Research  and  Sustainment  for  Crew  Systems  Interface  Laboratories  (R&SCSIL) 
contract  to  more  efficiently  support  AFRL/HEC.  For  example,  when  we  began  the  current 
effort,  the  incumbent  contractor  had  a  dozen  programmers  supporting  the  Synthesized 
Immersion  Research  Environment  (SIRE)  facility.  We  supported  the  vastly  more  complex 
Synthesized  and  Human  Aerospace  Forces  in  an  Immersion  Research  Environment 
(SAFIRE)  architecture  with  less  than  half  the  software  staff.  Here,  and  across  the  board,  we 
brought  on  highly  competent  engineers,  cut  out  layers  of  unnecessary  administration,  focused  our 
employees  on  the  R&SCSIL  work,  and  did  not  burden  them  with  corporate  projects.  We  also 
adapted  our  accounting  and  reporting  structure  to  better  correlate  with  the  Government's  research 
and  development  case  files.  Even  from  the  start,  when  the  incumbent  contractor  protested  the 
award,  SYTRONICS  stepped  up  with  strong  leadership  and  implemented  a  business  strategy,  at 
some  expense  to  the  company,  which  was  in  the  best  interest  of  the  laboratory  and  enabled  the 
work  to  continue  without  interruption. 

Our  R&SCSIL  work  clearly  demonstrated  SYTRONICS'  ability  to  provide  high-quality 
on-site  support  while  working  with  Government  technical  and  contracting  officials,  large  and 
small  business  subcontractors,  and  numerous  material  vendors.  SYTRONICS  also  maintained 
professional  and  personal  connections  with  many  important  Air  Force  and  DoD  agencies, 
including  the  Air  Combat  Command,  Air  Force  Special  Operations  Command,  National  Air  and 
Space  Intelligence  Agency,  Air  Force  Space  Command,  Air  Force  Battlelabs,  Missile  Defense 
Agency,  the  Naval  Air  Warfare  Center,  and  with  our  local  congressional  delegations. 
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SYTRONICS'  approach  for  R&SCSIL  was  to  provide  AFRL/HEC  technically  superior 
capabilities  and  resources  in  a  highly  cost-competitive  manner.  Our  proven  capabilities  include 
the  following. 


•  Highly  competent  and  creative  technical  staff  experienced  in  supporting 
AFRL/HEC  programs. 

•  Local  corporate  presence  and  resource-based  corporate  management  approach 
to  support  short-term  work,  surges,  and  lapses.  We  leveraged  both  on-site  and 
corporate-based  skill  sets  to  supplement  where  needed  for  short  suspense  studies, 
demonstrations,  and  projects.  For  example,  as  part  of  our  R&SCSIL  support  for 
the  AFRL/HEC  laboratories,  we  coordinated  corporate-based  software  support  for 
Collaborative  Research  in  Space  &  Temporal  Interfaces  Laboratory  (CR1STIL) 
and  Speech  Communication  Research  Engineering  Analysis  and  Modeling 
(SCREAM)  laboratories;  drafting,  electronics  and  hardware  technician  support  for 
Helmet-Mounted  Sensory  Technologies  (HMST);  and  mechanical  and  material 
support  for  Aerospace  Operations  Center  (AOC)  (now  Augmented  Time  Critical 
Analysis  and  Targeting  Facility  (aTCaTf)  on  an  as-needed  basis. 

•  Warfighter  experienced  employees— Many  members  of  our  staff  have  warfighter 
experience  bringing  a  warfighter's  perspective  and  understanding  to  AFRL/HEC 
support  and  linkage  to  many  DoD  organizations. 

•  Cradle-to-grave  military  acquisition  experience— we  understand  the  issues 
encountered  at  every  step. 

•  In-house  expertise  in  prototyping  and  fielding  flight-worthy  systems. 

•  Rapid  access  to  industry  and  academic  experts  through  fast,  low-cost  contracting 
vehicles. 


Historical  Perspective 

From  25  September  1998  through  30  September  2005,  SYTRONICS,  Inc.,  provided 
scientific  and  technical  support  to  the  Air  Force  Research  Laboratory,  Human  Effectiveness 
Directorate  for  the  R&SCSIL  Program,  Contract  F33615-98-D-6000. 

The  current  AFRL  Mission  and  Vision  statements  continue  the  theme  set  in  place  by 
Lieutenant  Colonel  Paul  M.  Fitts  when  the  AFRL  (then  the  Psychology  Branch  in  the  Aero 
Medical  Laboratory  at  Wright  Field)  was  established  in  May  of  1945.  The  statements  assert  that 
the  research  be  focused  on  improving  the  human-computer  interface  through  scientific  research. 
The  research  results  obtained  through  the  R&SCSIL  Program  help  to  improve  the  design  and 
development  of  warfighting  equipment.  These  achievements  preserve  Lt  Col  Fitts'  vision  for  the 
advancement  of  human  research  by  providing  engineers  with  the  scientific  data  necessary  to 
design  and  develop  equipment  that  meets  the  requirements  of  the  human  operator. 
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In  August  of  1998,  AFRL/HEC  funding  supported  a  combined  on-site  Logicon 
Technical  Services,  Inc.  (LTSI),  and  Veridian  staff  of  110+  personnel  performing  the  activities 
covered  by  the  R&SCSIL  contract.  By  November  of  1999,  52  personnel  members  were  hired  by 
SYTRONICS  to  support  the  program.  Twenty-five  LTSI  personnel  and  four  Veridian  personnel 
supported  the  contract  via  subcontract  agreements  with  SYTRONICS.  At  that  time,  a  total 
combined  staff  of  81— 68  full-time  and  13  part-time  personnel-supported  the  directorate. 

Organization  and  Laboratory  Support 

In  the  past,  Govemment/contractor  relations  for  technical  support  contracts  focused 
solely  on  the  research  as  a  collaborative  team  effort.  However,  today's  diminishing  research 
funding  and  the  need  for  more  outside  exposure  demand  that  the  entire  process  of  research, 
marketing,  and  technology  transition  be  collaborative  efforts  among  the  Government  and  support 
contractors.  The  main  program  management  thrust  at  the  onset  of  R&SCSIL  was  to  market  the 
laboratory  as  widely  and  effectively  as  possible  in  order  to  increase  the  use  of  their  existing 
resources,  broaden  the  useful  technology  base  and  operational  applications,  develop  more 
effective  Unmanned  Aerial  Vehicle  (UA  V)  and  space  vehicle  operator  interfaces,  provide 
increased  benefits  to  the  warfighter,  transition  technology  for  commercial  use,  and  increase 
annual  revenues  from  outside  sources.  SYTRONICS  leveraged  Small  Business  Innovation 
Research  (SBIR)  funding  to  accomplish  tasks  for  AFRL/HEC  (V oice-Head  Interface  Controller— 
VHIC;  Navy  Air-and  Bone-Conducted  Noise  Reduction  System)  and  AFRL/HEA  ( Aircrew 
Speech  Processor  (ASP)-,  Global  Horizontal  Operations  Support  Toolkit  (GHOST),  etc.). 
SYTRONICS  also  spearheaded  a  two-year  $4.1M  Congressional  Plus-Up  request  for 
FY05/FY06  to  establish  a  National  Center  of  Excellence  for  Networked  Warfighter  Decision 
Support  at  Wright-Patterson  Air  Force  Base  (WPAFB)  as  part  of  our  efforts  to  obtain  outside 
funding  for  AFRL/HEC.  This  funding  initiative  has  been  coordinated  with  the  Dayton 
Development  Coalition  and  Congressman  David  Hobson's  staff  to  provide  start-up  funding  for  a 
major  initiative  to  develop  and  study  the  human  factors  that  enhance  building  and  sustaining 
shared  situation  awareness  between  crew  members  performing  as  ground-based  and  airborne 
battle-managers;  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  asset  controllers;  air 
combat  pilots  during  Network-Centric  Operations;  and  in  particular  unmanned  vehicle  operators. 

SYTRONICS  also  spearheaded  an  effort  with  AFRL/HEC,  the  Airborne  Warning  and 
Control  System  (AWACS)  System  Program  Office  (SPO),  and  Boeing  to  gain  Office  of  the 
Secretary  of  Defense  (OSD)  funding  for  AFRL/HEC  transition  of  speech  technology  into 
AWACS,  and  leveraged  a  SBIR  match  to  double  the  funding  to  $960K. 

SYTRONICS  supported  AFRL's  Human  Effectiveness  Directorate  Warfighter  Interface 
Division's  five  branches:  System  Control  Interfaces,  Battlespace  Acoustics,  Collaborative 
Interfaces,  Battlespace  Visualization,  and  Cognitive  Systems.  There  were  15  individual 
laboratories  within  the  divisions.  They  include  the  following. 

•  Collaborative  Systems  Technology  Laboratory 

•  Crew  Aiding  and  IW  Analysis  Laboratory 

•  SCREAM  Laboratory 
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•  Aerospace  Vision  Experimental  Laboratory 

•  Color  Display  Laboratory 

•  Display  Test  and  Evaluation  (DT&E)  Laboratory 

•  Dynamic  Visual  Assessment  Facility 

•  Night  Vision  Operations  (NVO)  Laboratory 

•  Visual  Image  Evaluation  of  Windscreens  (VIEW)  Laboratory 

•  Visual  Symbology  Test  &  Evaluation  Facility 

•  Flight  Psychophysiology  Laboratory 

•  Advanced  UAV  Interface  Defense  Technology  Objective 

•  Multisensory  Overview  Large-scale  Tactical  Knowledge  Environment  Interface 
(MOLTKE)  Laboratory 

•  Research  on  Adaptive  Interfaces  for  Virtual  Environments  (RAIVE)  Laboratory 

•  SAFIRE  Facility 

Delivery  Order  Contract  Structure 

Thirteen  DOs  were  issued.  The  DO  0001  objective  was  to  provide  ongoing  and  sustaining 
technical  and  network  support  for  research  programs  in  the  AFRL/HEC  laboratories  and  facilities  as 
required.  The  required  work  included:  (1)  development  and  implementation  of  research  plans,  (2) 
sustaining  effort  of  AFRL/HEC  laboratories  and  facilities,  and  (3)  AFRL/HEC  computer  network 
support. 

DO  0002,  Visual  Display  Technology  (VDT),  provided  the  necessary  ongoing  research 
and  sustainment  of  the  Design  Parameters  for  Visually  Coupled  Systems,  NVOs,  VIEW,  and  the 
HMST  Programs.  The  scope  of  the  required  research  and  sustainment  included:  (1)  planning  and 
execution  of  exploratory  development  of  research  plans;  (2)  designing,  fabricating,  integrating, 
maintaining  documenting,  and  testing  of  VDT  hardware  and  software,  and  (3)  helmet  system 
certification  and  flight  testing. 

The  purpose  of  DO  0003,  Basic,  Exploratory,  and  Advanced  Development  of  Adaptive 
Interface  Technology,  was  to  provide  research  and  sustainment  for  the  ongoing  development  and 
evaluation  of  adaptive  interface  concepts  and  technologies  employing  combinations  of  virtual 
and  nonvirtual  techniques.  The  scope  of  the  required  work  included:  (1)  planning  and  execution 
of  a  combination  of  basic  research,  exploratory  development  and  advanced  development;  (2) 
sustaining  effort  of  the  human  interface  research  facilities  utilized  in  the  development  and 
evaluation  of  adaptive  interface  technology,  such  as  the  SIRE  and  Combat  Observation  and 
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Visualization  Environment  (COVE)',  (3)  perform  research  specified  by  the  US/French 
Memorandum  of  Understanding  (MOU)  regarding  the  Cooperation  Development  and 
Evaluation  of  Super  Cockpit  Technologies  including  hardware  and  software  development, 
systems  integration,  research  plan  development  and  execution,  and  documentation;  (4)  perform 
research  in  the  adaptive  interface  development  within  the  requirements  of  the  US/British  Vista 
Warrior  Project  Arrangement  (PA)  to  include  hardware  and  software  development,  systems 
integration,  research  plan  development  and  execution,  and  documentation;  (5)  perform  research 
as  specified  in  the  US/Australian  Virtual  Air  Commander  PA  to  include  hardware  and  software 
development,  systems  integration,  research  plan  development  and  execution,  and  documentation; 
(6)  perform  research  as  specified  in  the  Cooperative  Research  &  Development  Agreement 
(CRDA)  with  Kettering  Medical  Center  for  developing  interface  technology  applicable  to 
neurosurgery  to  include  hardware  and  software  development,  systems  integration,  research  plan 
development  and  execution,  and  documentation;  and  (7)  perform  research  for  the  New  World 
Vista  Program,  specifically  the  development  of  multisensory  and  adaptive  display  and  control 
concepts  and  their  application  to  Synthetic  Task  Environments  (STE)  such  as  the  Predator  STE. 

"Bioacoustic,  Noise,  and  Vibration  Research  and  Sustaining  Effort"  was  the  title  of  DO 
0004.  SYTRONICS  provided  ongoing  and  sustaining  research,  development,  and  technical 
efforts  for  the  research  programs  in  the  bioacoustic,  noise,  and  vibration  laboratories.  The  work 
included:  (1)  development  and  implementation  of  research  plans,  experimentation,  and 

prototypes;  and  (2)  the  sustainment  of  the  bioacoustic,  noise,  and  vibration  laboratories  and 
facilities. 

DO  0005,  Proof-of-concept  Prototype  Speech  Processing  System,  provided  for  research 
in  speech  technology  investigations  and  their  effectiveness  in  supporting  improved  warfighter 
training  during  air  operations  exercises.  The  purpose  of  this  work  effort  was  to  develop  and 
evaluate  a  proof-of-concept  prototype  speech  processing  system  that  could  manipulate  various 
types  of  information  in  the  speech  signal.  The  prototype  was  also  able  to  evaluate  the 
importance  of  that  information  for  speaker  identity  and  emotional  state  determination.  A  speech 
processing  system  was  evaluated  with  human  listeners  using  speech  collected  in-house  and  from 
standard  speech  databases. 

Vision  display  research  for  the  Strike  Helmet  21  (SH21)  Project  was  provided  under  DO 
0007.  This  DO  was  a  continuation  of  DO  0002,  VDT,  which  provided  the  necessary  ongoing 
research  and  sustainment  of  the  Design  Parameters  for  Visually  Coupled  Systems,  NVO,  VIEW, 
and  the  HMST  Programs. 

The  purpose  of  DO  0008,  Research  for  Multisensory  Interfaces  to  Optimize  Warfighter 
Performance,  was  to  provide  research  and  sustainment  for  the  ongoing  development  and 
evaluation  of  intelligent  multisensory  interface  concepts  and  technologies  employing 
combinations  of  virtual  and  non-virtual  techniques.  The  programs  supported  by  this  research 
include  Advanced  UAV  Interfaces,  Virtual  Air  Commanders,  Real-Time  Human  Engineering  for 
Unmanned  Combat  Air  Vehicle  (UCAV)  Automation,  and  Crew  Interlaces  for  Satellite 
Operations. 
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Under  DO  0009,  Research  to  Support  Information  Analysis  and  Exploitation, 
SYTRONICS  provided  research  and  sustainment  to  perform  analyses  and  modeling  of  human 
and  system  performance  to  support  acquisition  and  operation  decision-making.  The  programs 
supported  by  this  research  included  Biologically-Based  Signal  Processing  for  Information 
Operations;  Cognitive  Interfaces  for  ISR  Operations;  Cyber  Crew  Interface  Development; 
Collaborative  System  Technology;  User  Modeling  and  Intelligent  Aiding;  Adversarial  Decision 
Modeling;  Cognitive  Systems  Engineering;  and  Rapid  Insertion  Technologies. 

In  support  of  DO  0010,  Research  to  Support  Visual  Display  Systems,  SYTRONICS 
provided  research  and  sustainment  to  enhance  warfighter  vision  and  visual  interfaces  to  improve 
weapon  system  effectiveness  during  day  and  night  operations.  The  programs  supported  by  this 
research  included  Aerospace  Displays  Program,  Crew  Station  Avionics,  High  Definition  Display 
Technology  Program,  Organic  Light  Emitting  Diode  Dual  Use  in  Science  and  Technology,  and 
Windscreen  Visual  Interface  Effects. 

The  purpose  of  DO  0011,  Research  for  Defense  Advanced  Research  Projects  Agency 
(DARPA)  Augmented  Cognition  Program,  was  to  develop  and  apply  real-time  operator  cognitive 
state  assessment  technology  to  the  UCA  V Decision  Aiding  System  (UDAS)  under  the  Real-Time 
Human  Engineering  for  UCAV  Automation  Program  collaboration  with  DARPA.  This  effort 
provided  research  and  sustainment  for  the  ongoing  development  and  evaluation  of  technologies 
to  assess  in  real-time  the  cognitive  state  of  operators  controlling  UAVs,  and  to  evaluate  operator 
decision  and  performance  aiding  techniques  that  are  enabled  by  the  real-time  state  assessment. 

The  purpose  of  DO  0012,  Research  to  Support  Information  Operation  and  Air  Operation 
Centers,  was  to  provide  analyses,  technology  development,  and  modeling  of  human  and  system 
performance  to  support  acquisition  and  operation  of  decision-support  technologies.  The 
programs  supported  by  this  research  included  Cyber  Crew  Interface  Development,  Visualization 
and  Decision  Aiding  for  Air  Operation  Centers,  User  Modeling  and  Intelligent  Aiding, 
Adversarial  Decision  Modeling,  and  Cognitive  Systems  Engineering. 

DO  0013  provided  for  support  to  the  Research  for  Strike  Warrior  PA.  This  arrangement, 
PA  DoD-MoD-AF-0 1-0003,  was  one  facet  of  the  DoD-MoD  Research  and  Development 
Projects  (RDP)  MOU  between  the  United  States  DoD  and  the  United  Kingdom  Ministry  of 
Defense  (MoD).  One  component  of  the  Strike  Warrior  PA  was  collaboration  on  interface  issues 
for  aircraft  pilots,  Command  and  Control  (C2)  operators,  and  UAV  and  UCAV  operators.  This 
effort  provided  research  and  sustainment  for  the  ongoing  development  and  evaluation  of 
technologies  to  use  advanced  interface  technology  to  improve  human  performance,  understand 
and  improve  human  interaction  with  autonomous  systems,  and  assess  in  real-time  the  cognitive 
state  of  operators  controlling  UAVs,  and  to  evaluate  operator  decision  and  performance-aiding 
techniques  that  are  enabled  by  the  real-time  state  assessment. 
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PROGRAM  MANAGEMENT 


SYTRONICS  has  a  very  proactive  management  philosophy.  We  are  vigilant  for  cost, 
schedule,  performance,  and  personnel  problems  and  aggressively  pursue  alleviating  them.  For 
example,  lessons  learned  early  in  the  R&SCSIL  Program  resulted  in  many  significant  changes. 
We  improved  our  human  resources  process  by  consulting  with  an  independent  firm  specializing 
in  human  resources  and  implementing  many  recommended  improvements.  We  changed  our 
program  management  structure  and  transitioned  a  laboratory  Ph.D.  with  13  years  experience  in 
AFRL/HEC  to  serve  in  the  capacity  as  technical  lead  and  program  manager.  Management  also 
reevaluated  subcontracting  emphasis  and  customer  coordination  which  resulted  in  overall 
improvement  in  the  R&SCSIL  Program. 

In  support  of  AFRL/HEC,  we  developed  program  management  plaas  as  required  that 
included  task  breakdowns,  schedules,  budgeting,  resource  loading,  equipment  utilization,  and 
facility  needs.  Our  program  management  structure  shortened  the  decision-making  process, 
empowered  the  DO  Leads  to  coordinate  directly  with  the  Government,  and  provide  deployed 
decision-making.  They  could  ensure  the  program  was  properly  executed  by  coordinating  all 
requirements,  assigning  the  appropriate  staff  (with  Government  guidance),  providing  necessary 
resources,  monitoring  the  work,  reviewing  and  updating  plans,  adjusting  to  changing 
requirements,  and  properly  documenting  and  reporting  the  project  status  and  deliverables. 
Finally,  through  a  standard  set  of  weekly  management  meetings,  monthly  technical  and  quarterl  y 
management  reviews,  we  ensured  the  best  quality  support  to  the  Government. 

STAFFING 

SYTRONICS  employed  over  50  full-  and  part-time  employees  solely  supporting 
AFRL/HEC's  R&SCSIL  effort.  We  reshaped  the  workforce  to  accommodate  the  evolving 
requirements  and  adapt  the  staff  as  needed  to  provide  the  optimal  cost-effective  mix  of  technical 
capabilities. 

R&SCSIL  required  mostly  low-to-mid  level  technical  support.  As  shown  in  Figure  1, 
90%  of  our  R&SCSIL  Scientific  and  Engineering  (S&E)  staff  was  equivalent  to  DR1  and  DR2 
level  civil  servants  as  defined  by  the  Laboratory  Demonstration  model.  In  addition  to  providing 
these  engineers,  programmers,  and  scientists,  we  also  provided  graphics  support,  Precision 
Measurement  Equipment  Laboratory  (PMEL),  accountable  equipment  management,  corporate- 
based  shipping  and  receiving,  Scientific  and  Technical  Information  (STINFO)  support,  and 
subject  pool  management  (with  a  permanent  staff  of  10  part-time  subjects). 
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Figure  1.  Technical  Support 

SYTRONICS  continually  worked  with  the  Government  to  determine  the  best  mix  of  staff 
to  maximize  technical  support  and  cost-effectiveness  without  harming  short  or  long  term 
productivity.  When  a  new  position  became  available,  resumes  were  provided  for  a  pool  of 
candidates.  Our  criteria  was  to  provide  the  best  candidate  for  every  position  based  on  desired 
skill  set,  education,  and  past  experience.  We  strove  to  have  well-informed  and  motivated 
employees  who  knew  what  was  expected  of  them,  how  to  resolve  any  issues  that  arose,  and 
strove  to  perform  at  their  utmost  every  day.  This  was  evidenced  by  our  low  turnover  rate  which 
maintained  laboratory  continuity. 

In  an  effort  to  further  increase  the  cost-effectiveness  of  our  staff,  we  coordinated,  where 
possible,  to  provide  co-op  students  and  interns  to  supplement  the  workforce  with  young, 
enthusiastic,  and  cost-effective  labor  resources.  These  interns  and  co-ops  were  placed  with  the 
Government  at  a  reduced  overhead  rate.  The  use  of  co-ops  and  interns  not  only  provided 
talented,  cost-effective,  part-time  support  but  also  allowed  the  Government  to  preview,  select, 
and  groom  possible  future  full-time  personnel.  Under  R&SCSIL,  SYTRONICS  was  able  to 
bring  in  an  intern  to  SCREAM  Laboratory  and  he  was  later  hired  full-time— first  as  a  contractor 
and  then  as  a  civil  servant  in  AFRL/HECP.  Similarly,  another  SYTRONICS  employee  was  first 
hired  as  an  intern  and  then  became  full-time  supporting  AFRL/HECB.  Also,  a  graduate  student 
was  hired  as  a  part-time  employee  to  develop  and  complete  his  Ph.D.  dissertation  at  The  Ohio 
State  University  (OSU)  in  coordination  with  research  goals  in  the  SCREAM  Laboratory. 

MANAGERIAL  REPORTING 

Eighty-seven  Monthly  Status  Reports,  41  Final  Technical  Reports,  159  Itemized  Labor 
Costs  Status  Reports,  and  27  Funds  and  Manhour  Cost  Reports  were  submitted  throughout  the 
duration  of  the  contract.  Quarterly  Program  Reviews  (QPRs)  were  conducted  at  the 
Government  work  site  throughout  the  contract  term.  SYTRONICS  projects  were  carefully 
correlated  to  the  AFRL/E1EC  6.2  Roadmaps  to  ensure  compliance  with  Government  goals. 
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OPERATIONS  MANAGEMENT 


For  administrative  and  logistical  support,  SYTRONICS  offered  large  business  capability, 
but  with  small  business  flexibility.  SYTRONICS  provided  efficient  low-cost  subcontracting. 
Defense  Contract  Audit  Agency  (DCAA)- approved  financial  accounting,  purchasing,  and  fast 
and  flexible  travel  services.  Our  contracts,  accounting,  and  purchasing  departments  received 
numerous  high  marks  and  compliments  for  their  speed,  efficiency,  and  flexibility.  We  worked 
with  the  Government  to  provide  customized  financial  and  technical  reports  that  met  individual 
laboratory  managers'  needs. 

SYTRONICS  implemented  an  improved  management  scheme  that  it  promised  in  its 
contract  proposal.  Rather  than  use  multiple  dedicated  managers  as  prior  contractors  had,  we 
appointed  DO  Lead  Engineers  to  lead  the  technical  areas  and  supervise  supporting  personnel. 
This  supervisory  role  had  minimal  interference  with  the  supervisor’s  primary  technical  duties. 
The  DO  Leads  worked  with  the  individual  Government  laboratory  managers  and  the  program 
manager  to  supervise  and  help  manage  the  laboratory  contractor  staff  to  ensure  the  right  people 
were  available  at  the  right  time  to  get  the  job  done.  The  DO  Leads  were  also  responsible  for 
applying  new  capabilities  as  required  by  the  Government  leads.  SYTRONICS  corporate 
philosophy  is  to  let  project  personnel  concentrate  on  their  designated  tasks. 

This  approach  minimized  management  overhead,  provided  some  career  progression  for 
technical  personnel  while  preserving  their  technical  experience,  and  put  technical  issues  at  the 
forefront  of  program  management.  This  arrangement  resulted  in  management  better  focused  on 
science  and  engineering  productivity  at  a  net  lower  cost  to  the  Government.  SYTRONICS 
believes  this  arrangement  resulted  in  better  technical  management  and  a  considerable  savings  of 
man-years  compared  to  dedicated  managers  over  the  five  years  of  the  contract. 

Other  management  savings  were  achieved  by  merging  the  purchasing  function  with 
SYTRONICS'  existing  materials  procurement  capability.  The  dedicated  base  purchasing  staff 
was  merged  with  other  company  functions  so  that  the  cost  of  the  personnel  was  shared  between 
SYTRONICS'  other  purchasing  activities  and  the  contract.  This  also  resulted  in  lower 
purchasing  costs  to  the  Government.  One  of  the  purchasing  personnel,  who  had  a  significant 
health  problem,  was  able  to  continue  working  until  the  problem  necessitated  going  on  disability. 
The  situation  was  managed  by  SYTRONICS  so  there  was  no  reduction  in  services  while  showing 
the  employee  the  utmost  in  consideration  for  her  health  problems. 

Another  accommodation  to  the  Government's  way  of  doing  business  was  in  the  area  of 
contract  reporting.  SYTRONICS  adapted  its  monthly  progress  and  financial  reporting  to  match 
the  Government's  requirements  for  its  own  reporting.  This  enabled  the  Government  contract 
manager  to  "cut-and-paste"  SYTRONICS'  reports  directly  into  the  Government's  reports.  This 
saved  time  and  resources  on  the  Government  side  without  any  additional  contract  costs  and 
allowed  smoother  information  sharing  between  the  Government  and  SYTRONICS  management 
functions.  This  sort  of  accommodation  is  the  hallmark  of  SYTRONICS'  interaction  with  the 
Government. 
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A  similar  format  accommodation  was  made  with  regards  to  technical  reports. 
SYTRONICS  adopted  the  Government  technical  report  format  for  its  reporting,  allowing  easy 
transition  of  those  internal  reports  into  Government  publication  format.  The  SYTRONICS 
reports,  delivered  in  electronic  format,  were  submitted  virtually  unchanged  as  Government 
technical  reports,  with  a  considerable  savings  in  document  preparation  costs  for  the  Government. 

SYTRONICS  also  supported  industry  efforts  of  common  interest  with  the  Government, 
contributing  personnel  and  fiscal  support  to  the  International  Symposium  on  Aviation 
Psychology  when  it  was  held  in  Dayton,  OH  in  2003.  SYTRONICS  was  a  contributor  to 
sponsoring  the  meetings  and  its  personnel  served  on  the  organization  and  technical  committees. 

SYTRONICS  also  established  travel  policies  and  procedures  for  worldwide  travel  and 
experimental  support.  On  R&SCSIL,  we  supported  studies  at  numerous  remote  locations  in  the 
United  States  as  well  as  Europe  and  Australia.  All  travel  was  accomplished  in  accordance  with 

the  Joint  Travel  Regulation  (JTR). 

Finally,  SYTRONICS  also  ensured  compliance  with  all  Air  Force  and  WPAFB  computer 
access  and  security  requirements  and  fully  complied  with  the  "Computer  and  Network  Access 
Requirements  for  User's  and  Equipment"  and  all  rules  and  regulations  regarding  Automated 
Data  Processing  Equipment  ( ADPE ). 

FINANCIAL  MANAGEMENT 

SYTRONICS  was  one  of  the  first  contractors  in  Dayton  to  use  the  Defense  Finance  and 
Accounting  Service's  (DFAS)  Web  Invoicing  System  (WInS)  back  in  1998.  This  allowed  us  to 
use  many  DFAS  internet-based  tools  (such  as  the  Contractor  Invoice  System  (COINS)  and 
Mechanization  of  Contract  Administration  Services  (MOCAS)  tool)  to  submit  invoices,  track 
payments,  close  out  DOs,  etc.,  with  the  Government. 

SYTRONICS  went  to  considerable  trouble  to  modify  its  accounting  practices  to  make 
them  compatible  with  the  financial  reporting  required  by  the  Government.  All  the  Government 
needed  to  do  was  request  a  change,  and  the  local  staff  were  able  to  accommodate  the 
Government  managers  with  very  little  delay  or  cost.  This  flexibility  in  accounting  was 
something  the  Government  exercised  often  and  the  results  improved  overall  contract 
management. 

Minimizing  overhead  costs  paid  considerable  dividends  to  the  Government  in  terms  of 
reduced  contract  costs.  As  for  overhead  rates  over  the  entire  term  of  the  contract,  rate  increases 
over  the  five  years  of  the  contract  were  modest  and  deemed  acceptable  to  the  Government. 

As  a  small  business,  we  handled  material  purchasing  and  procurement  with  speed, 
efficiency,  and  flexibility.  On  R&SCSIL,  we  managed  an  average  of  over  450  purchases  (over 
$600K)  per  year.  For  large  and  small  dollar  purchases,  special  test  equipment,  calibration, 
repair,  and  other  needs,  SYTRONICS  utilized  our  proven  procurement  processes  including 
signature  approval,  vendor  selection  based  on  price/availability,  Purchase  Order  (PO) 
processing,  accounting,  shipping  and  receiving,  inspection,  and  delivery.  For  software  and  other 
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purchases  requiring  a  Computer  System  Requirement  Document  (CSRD),  we  ensured  that  all 
purchases  complied  with  CSRD  and  software  registration  regulations  and  that  all  software  was 
delivered  first  to  the  local  Designated  Approval  Authority  (DAA)  for  registration  before  being 
delivered  to  its  final  destination.  Finally,  we  ensured  the  appropriate  equipment  was  processed 
for  any  ADPE  requirements.  In  addition  to  the  standard  procurement  process,  we  provided  a 
well-documented  method  for  using  and  accounting  for  petty  cash  for  small  (under  $100) 
purchases.  Our  purchasing/accounting  system  complied  with  all  state  and  federal  tax  and 
procurement  regulations. 


SUBCONTRACT  MANAGEMENT 

SYTRONICS  promised  in  its  proposal  to  subcontract  in  a  low-cost,  easy  access,  and  fast- 
reaction  time  fashion  with  whomever  the  Government  chose  for  technical  work.  On  R&SCSIL, 
we  subcontracted  over  120  subcontracts  to  over  80  businesses  of  all  sizes— including  large 
multinational  defense  corporations  such  as  Boeing,  Lockheed  Martin,  Northrop  Grumman, 
General  Dynamics,  SAIC,  BAE,  and  SRA  International— and  academic  institutions  including  the 
Georgia  Tech  Research  Institute,  OSU,  Washington  University,  University  of  Toronto,  and  over 
a  dozen  others. 

Many  of  the  subcontracts  were  let  to  small  companies  and/or  individuals  that  had  no 
previous  experience  in  doing  business  with  the  Government.  SYTRONICS  provided  guidance  to 
those  companies  and/or  individuals  to  expedite  the  subcontracting  process.  Larger  businesses 
often  needed  to  be  carefully  watched  (fiscally)  when  receiving  relatively  small  subcontracts  to  do 
technical  work.  SYTRONICS  diligently  managed  their  bureaucratic  inertia,  prompting  regularly 
for  bills  and  clearing  them  quickly  to  manage  the  expenditure  of  funding  so  important  to  contract 
management  and  fiscal  resource  protection.  Subcontract  rates  were  diligently  negotiated  with 
the  companies  involved  and  several  times  SYTRONICS  was  able  to  get  them  reduced. 

Sole  Source  contracts  were  usually  established  in  less  than  five  days  from  the  time  the 
need  had  been  identified.  Competitively  bid  subcontracts  were  usually  established  in  less  than 
three  weeks  including  solicitation  and  evaluation.  After  a  subcontract  was  awarded,  any 
reporting  requirements  and  deliverables  were  then  coordinated  through  SYTRONICS  and 
subcontractor  performance  was  monitored  without  undue  burdening.  Standard  subcontracts 
simply  required  monthly  status  reports  and  monthly  invoices,  in  addition  to  the  subcontract 
product. 

Upon  completion  of  all  deliverables  and  Government  approval,  the  subcontract  was 
closed  out  and  any  remaining  funds  were  de-obligated  and  returned  to  the  Government. 
Subcontracts  were  finalized  by  obtaining  the  appropriate  closeout  documents,  the  final  invoice, 
and  the  Government's  signature  on  a  database-generated  form,  the  Subcontractor  Deliverable 
Review  Form,  created  especially  for  this  unique  program.  This  form  allowed  the  Government 
customer  to  interface  directly  with  the  subcontractor,  yet  allowed  SYTRONICS  to  coordinate  the 
successful  completion  of  the  effort  with  minimal  disruption  to  the  customer.  The  subcontract 
database  also  reduced  the  time  required  to  prepare  a  subcontract  agreement  by  at  least  50%. 
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Seven  competitive  subcontracting  opportunities  were  identified  over  the  term  of  the 
contract,  two  of  which  were  cancelled.  The  remaining  five  subcontracts  were  issued  after  a 
competitive  process.  SYTRONICS  managed  the  competitive  bidding,  evaluated  the  bids  in  an 
expeditious  fashion,  and  enabled  the  Government  to  get  the  best  possible  performers  without 
significant  time  delay  or  increased  cost. 


TECHNICAL  TASKS  AND  ACCOMPLISHMENTS 


Myriad  technical  accomplishments  were  achieved  through  research  conducted  by  the 
combined  talents  and  expertise  of  the  AFRL,  SYTRONICS,  and  subcontractor  scientists  and 
engineers.  A  list  of  technical  reports  completed  during  the  course  of  the  R&SCSIL  contract  is 
provided  in  Appendix  A.  Following  is  a  summary  of  some  of  the  technical  achievements  that 
SYTRONICS  accomplished,  or  played  an  important  role  in  facilitating  what  the  Government 
accomplished. 


TASKS 


The  nature  of  research  support  under  R&SCSIL  followed  a  paradigm  as  diagrammed  in 
Figure  2.  The  support  was  broken  down  into  a  series  of  tasks  including  the  following. 


Research  Planning 
Experiment  Planning 

Implementation,  Analysis,  and  Documentation  of  Experiments 
Prototype  Development  and  Demonstrations 


We  will  look  at  SYTRONICS'  support  under  each  of  these  areas  as  well  as  general  support 
functions  such  as  equipment  calibration,  media  shop,  machine  shop,  etc. 


WARFIGHTER  VALIDATION 


Open  PUBLISHABLE  RESULTS 
Literature  *  When  Security  Permits  I 


Research 

Planning 


u 


REPLANNING 


FEEDBACK 


Figure  2.  Research  Support  Paradigm 
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Research  Planning 


The  nature  of  research  planning  is  to  provide  researchers  with  a  guide  to  the  progression 
of  work  in  a  particular  research  area.  It  focuses  on  program  goals  and  provides  an  organized, 
stepwise  flow  of  how  the  research  should  proceed.  The  research  plan  is  a  living  document, 
which  is  updated  based  on  fmdings-to-date,  current  literature,  and  existing  AFRL/HEC 
roadmaps. 

The  scope  of  research  planning  under  R&SCSIL,  aside  from  obviously  supporting  all 
laboratories  and  laboratory  operations,  was  to  adapt  and  evolve  to  match  the  changing  nature  of 
research  in  AFRL.  A  paradigm  shift  has  been  underway  in  military  laboratory  funding.  Long¬ 
term  (6.2)  development  programs  are  giving  way  to  shorter  horizon  (two-three  year  6.2)  efforts 
and  to  more  (6.3)  advanced  development  efforts.  Heavy  emphasis  is  on  quickly  transitioning 
from  experimental  science  to  prototype  implementations,  suitable  for  testing  on  airframes  and  in 
other  operational  settings.  Warfighter  input  and  support  must  be  sought  out  early  and  often 
during  research  planning,  experimental  design,  experimentation,  and  prototype  development  in 
order  to  transition  proven  technologies. 

The  technical  approach  to  research  planning  is  comprised  of  three  distinct  stages:  initial 
planning,  update  and  maintenance  of  the  plan,  and  human  use  protocols. 

Initial  Planning:  For  initial  planning,  SYTRONICS  worked  with  Government  scientists 
and  engineers,  as  well  as  warfighters  (through  operational  units  or  the  Battlelabs)  to  develop  the 
plan  by  providing  whatever  specific  technical  expertise  (design  and  analysis,  subject  matter 
experts  (SMEs),  consultants,  theoretic  science,  literature  reviews,  etc.)  deemed  necessary.  Plans, 
in  the  form  of  documents,  briefing  slides,  or  both,  were  submitted  (usually  informally)  to  the 
Government  for  approval.  Subsequently,  SYTRONICS  provided  configuration  and  revision 
control  of  the  research  plan  because  the  inevitable  changes  need  to  be  carefully  tracked. 

Monitor.  Develop,  Update,  and  Maintain  the  Research  Plan:  One  of  SYTRONICS' 
outstanding  capabilities  is  to  research  and  integrate  new  commercial  off-the-shelf  (COTS) 
hardware  and  software  to  maximize  results  with  minimal  investment.  As  R&SCSIL  support 
contractor,  we  continued  to  monitor  trade  journals,  attend  professional  society  meetings,  interact 
with  other  Government  agencies,  and  perform  Web  and  literature  reviews  in  order  to  stay  current 
on  the  latest  technology  and  ideas  and  integrate  new  ideas  into  the  laboratory  research  planning, 
resulting  in  the  most  cost-effective  innovative  solutions  to  meet  research  requirements. 

Human  Use  Protocols:  Safety  is  always  paramount  in  SYTRONICS'  research;  we 
understand  the  importance  of  Institutional  Review  Board  (IRB)  protocols  for  the  protection  of 
research  participants  and  the  compliance  with  Air  Force  regulations.  As  part  of  our  R&SCSIL 
support,  we  made  sure  we  knew  the  regulations  and  we  supported  AFRL/HEC  in  preparing, 
writing,  and  briefing  IRB  protocols.  Numerous  SYTRONICS  employees  took  part  in  IRB 
protocols  in  support  of  NVO,  Aerospace  Vision  Program  (AVX),  Bioacoustics,  SAFIRE,  and 
other  laboratories.  Since  IRB  approval  is  necessary  for  all  human  use  studies,  early  protocol 
submission  was  a  necessary  part  of  the  planning  process.  Long  lead  times  for  Surgeon  General's 
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office  approval  means  protocols  must  be  submitted  well  in  advance  of  the  proposed  studies.  No 
SYTRONICS  supported  study  was  ever  delayed  because  it  lacked  IRB  approval. 

Experimental  Planning 

While  research  planning  shapes  the  general  approach  to  conducting  studies,  the  nature  of 
the  next  step  in  the  process,  experimental  planning,  is  to  deal  with  the  details  of  creating  specific 
studies  with  sufficient  control,  validity,  and  reliability  to  resolve  the  truthfulness  of  specific 
hypotheses.  At  the  same  time,  consideration  to  the  statistical  analysis  of  results  must  be  given  so 
that  the  desired  analyses  are  possible  and  have  sufficient  statistical  power  to  give  the  analysis  a 
fair  opportunity  to  succeed.  The  specialized  nature  of  experiments  within  the  AFRL/HEC 
facilities  calls  for  experience  to  understand  the  nuances  of  each  research  area  to  effectively 
define  the  scope  of  the  planning.  SYTRONICS’  staff  contains  experienced  experimental 
scientists  who  provided  or  assisted  in  experimental  planning  for  AFRL/HEC  research. 

For  example,  two  types  of  research  are  conducted  by  the  Battlespace  Visualization 
Branch  -  characterization  research  and  human  factors  psychophysical  research.  Characterization 
research  deals  with  the  test  and  evaluation  of  visual  displays.  Night  Vision  Devices  (NVDs), 
aircraft  transparencies,  and  basic  human  vision  under  laboratory  conditions,  using  standardized 
test  batteries.  Human  factors  research  deals  with  perceptual  performance  using  or  looking 
through  displays,  transparencies,  or  usability  of  visual  augmenting  technologies  like  NVDs. 

Each  of  these  areas  had  different  and  specific  experimental  planning  scope  and  needs.  In 
other  cases,  standards  did  not  exist  so  new  test  methods  were  developed  to  answer  research 
hypotheses  or  user  questions.  Some  in-house  methods  developed  by  the  Government  and 
SYTRONICS  engineers  have  been  adopted  by  industry.  Examples  of  these  are  American 
Society  for  Testing  and  Materials  (ASTM)  test  methods  and  a  procedure  for  determining  Night 
Vision  Goggle  (NVG)  cockpit  compatibility  in  civilian  aircraft. 

Our  statistical  approach  to  studies  applied  designs  such  as  repeated  measures ,  interrupted 
time  series,  or  split-plot  designs  which  are  condition-,  time-  and  subject-efficient.  By  carefully 
choosing  a  minimum  number  of  independent  variables,  we  could  allow  interpolation  of  findings 
to  cover  the  range  of  possibilities,  while  testing  fewer  conditions.  Employing  multiple 
dependent  measures  (objective  and  subjective),  and  consulting  with  SMEs  to  develop  new 
measures  that  could  better  capture  the  independent  variables'  effects,  helped  to  enhance  research 
outcomes.  Sometimes  measures  were  embedded  in  prototype  software  to  permit  automated  data 
collection  in  the  laboratory  and  the  field.  This  method  is  particularly  effective  in  cognitive 
systems  engineering,  where  additional  objective  measures  are  needed  to  supplement  the 
commonly-used  subjective  measures. 

Because  of  its  diversity,  each  of  the  R&SCSIL  technical  areas  had  very  different  and 
specific  experimental  planning  needs  and  therefore,  required  different  approaches.  For  example, 
characterization  research  of  displays  in  the  DT&E  Laboratory,  NVDs  in  the  NVO  Laboratory 
and  aircraft  transparencies  in  the  VIEW  Laboratory  took  several  forms,  and  our  approaches  did 
also.  If  reevaluation  of  a  system  was  required,  we  would  review  previous  evaluations  and 
determine  the  appropriate  battery  of  tests,  available  fixtures  and  test  rigs,  and  equipment  setup 
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required  to  obtain  data  repeatability.  Prior  to  developing  new  prototype  evaluation  devices,  we 
would  conduct  or  update  literature  reviews  as  required  to  prevent  duplicating  prior  work,  verify 
approach  soundness,  and  ensure  testing  methodologies  and  specifications  were  valid.  All  of 
these  studies  required  critical  alignment  and  proper  positioning  as  well  as  the  setting  of 
appropriate  calibrated  lighting  levels.  Since  DT&E,  VIEW,  and  NVO  receive  warfighter  assets 
for  rapid  evaluation,  SYTRONICS  was  responsive  to  often  demanding  schedules.  Often,  time 
constraints  for  these  evaluations  did  not  always  allow  for  in-depth  planning.  In  those  cases, 
SYTRONICS  would  use  our  extensive  experience  in  rapid-response  testing  and  evaluation  in  the 
VIEW  and  NVO  laboratories  to  precisely  measure  the  asset,  reduce  and  analyze  the  data,  write 
the  report  and  assist  in  returning  the  asset  to  the  field  in  a  timely  fashion. 


Figure  3.  Display  Test  and  Evaluation  Laboratory 

Psychophysical  research  studies  (visual,  auditory,  and/or  haptic)  had  different  objectives 
and  requirements.  Here,  we  first  defined  the  specific  tasks  for  which  performance  improvements 
were  desired  (visual  acuity,  target  acquisition,  auditory  localization,  etc.),  then  specified  the 
parameters  that  were  expected  to  influence  operator  performance  (color,  contrast,  sound  level, 
distortion,  etc.),  as  well  as  the  experimental  design  and  statistical  variables.  We  determined 
environmental  variables  and  employed  optimum  values  of  parameters  for  performance  of  tasks  in 
specific  environments  to  test  performance.  For  example,  visors,  laser  eye  protection,  filters, 
NVDs,  displays,  etc.,  may  be  obtained  and  the  displays  characterized  with  and  without  these 
devices  for  quantification  of  effects  on  the  visual  perception  of  color  displays. 

Once  experimental  tasks,  variables,  and  parameters  were  defined,  we  wrote  and 
coordinated  with  the  Government,  a  detailed  experimental  design  that  reflected  the  research  plan, 
based  on  appropriate  statistical  analysis,  to  ensure  the  study  would  provide  meaningful  and 
statistically-valid  data.  For  this  plan,  we  also  determined  subject  requirements  and  coordinated 
the  screening  of  subjects.  The  plan  defined  how  acquired  equipment  and  systems  would  be 
determined,  set  up,  calibrated,  tested  and  documented  (in  writing  as  well  as  photographically), 
and  how  data  collection  software  or  hard  copy  data  sheets  would  be  written  as  required, 
depending  on  the  nature  and  laboratory  conducting  the  study.  When  prototype  displays  or  NVDs 
were  field  evaluated,  our  plan  specified  how  we  assisted  the  Government  in  scheduling 
appropriate  facilities,  aircrews,  or  aircraft  for  testing. 
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Defining  the  specific  parameters  of  an  experiment  to  answer  research  hypotheses  always 
has  its  risks.  To  mitigate  such  risk,  SYTRONICS’  primary  aid  in  experimental  planning  was  the 
use  of  "pilot"  studies.  Such  studies  run  a  few  subjects  in  advance  of  the  "full"  study  to  test 
experimental  procedures,  shake  down  apparatus,  and  estimate  statistical  parameters.  Use  of  pilot 
studies  added  a  small  cost  to  overall  conduct  of  the  research,  but  prevented  wasting  the  larger 
experimental  effort  due  to  procedural  problems,  artifact  interference,  and  confounding  variable 
intrusion.  Furthermore,  we  used  estimates  of  group  mean  separation  to  estimate  statistical  power 
of  the  main  design  and  adjust  sample  size  to  most  efficiently  achieve  the  desired  power  level. 

Experimental  Research 

Conducting  high-relevance,  focused,  military  experimental  research  is  what  sets  AFRL 
and  the  other  military  laboratories  apart  from  universities  and  industry.  Universities  relish  doing 
"military-related"  activities,  but  prefer  to  frame  the  military  issue  in  lower-relevance,  more 
general  tasks.  Industry  often  lacks  continuity  of  effort,  dependent  on  episodic  funding  from 
military  sources.  The  Battlelabs  do  high-relevance  demonstrations,  but  have  not  been  adept  at 
scientific  data  collection.  Whether  it  is  to  develop  technology  for  warfighters,  or  to  answer  a 
mission  critical  question,  data  collection  and  analysis  is  what  distinguishes  the  AFRL  military 
research  mission.  The  AFRL/HEC  organization  has  a  long  and  distinguished  history  of 
innovative  science,  technology  development,  and  successful  transition  to  the  warfighter. 

The  new,  evolving  nature  of  military  laboratory  research  makes  the  nature  of  executing 
studies  more  demanding  than  in  typical  university  research.  The  usual  requirements  for  strict 
control  of  confounding  variables,  careful  administration  of  independent  variables,  errorless  data 
collection,  and  reliable  procedures  still  apply.  Additional  burdens  including  high  external 
validity,  dependency  on  prototype  devices  and  interfaces,  differences  between  naive  subjects  and 
unique  military  operators,  and  carry-over  effects  from  existing  systems  or  standard  operating 
procedures  further  complicate  AFRL  research.  Field  data  collections  are  often  constrained  by 
operational  priorities,  turning  these  studies  into  quasi-experiments.  Researchers  must  be 
prepared  to  deal  with  known  confounding  variables  to  determine  the  real  cause  and  effect 
relationships. 

The  scope  of  R&SCSIL  included  the  five  primary  areas  of  research  in  AFRL/HEC: 
Battlespace  Acoustics,  Battlespace  Visualization,  Cognitive  Systems,  Collaborative  Interfaces, 
and  System  Control  Interfaces.  Each  area  and  laboratory  within  the  area  conduct  experimental 
research  tailored  to  their  objectives  and  methods,  common  to  their  particular  pursuit. 
SYTRONICS  supported  this  experimental  research  as  we  describe  below. 

The  first  step  for  all  studies  was  to  collect  pilot  data  once  the  apparatus  was  fully 
operational.  The  initial  subjects  for  the  pilot  studies  may  have  been  the  technical  developers 
themselves  or  other  laboratory  personnel  and  these  studies  served  to  "shake  out"  the 
experimental  procedures.  In  high  cost  or  mission  critical  studies,  subjects  from  the  actual  subject 
pool  may  have  been  used  to  ensure  pilot  data  accuracy.  Outcomes  could  be  used  to  estimate 
group  parameters  and  determine  the  statistical  power  of  the  planned  analysis,  changing  group 
size  and  statistical  tools  based  on  the  pilot  subjects.  The  whole  process  was  documented  as  the 
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introduction  and  methods  for  subsequent  publications  and  to  verify  the  apparatus  and  methods  if 
the  work  is  ever  challenged. 

The  diversity  and  breadth  of  laboratories  within  AFRL/HEC  make  it  impossible  to  cover 
all  aspects  of  experimental  research  in  this  writing;  however,  here  we  provide  specific  examples 
of  some  of  our  support. 

A  recent  evaluation  of  advanced  AWACS  displays  involved  data  collection  in  the 
MOLTKE  Laboratory  with  mission-ready  combat  air  controllers.  During  the  experiment 
development,  SYTRONICS  staff  and  SMEs  ran  as  pilot  subjects  to  debug  experimental  software 
and  estimate  tire  size  of  the  effect.  In  the  subsequent  study,  real  AWACS.  crewmen  directed 
semi-automated  aircraft  to  evaluate  interface  technologies  like  speech  recognition  and  advanced 
display  formats  for  data  fusion.  This  study  is  a  good  example  of  automated  data  collection  in 
which  the  display  software  was  instrumented  to  collect  performance  data  during  the  execution  of 
simulated  air  intercepts  directed  by  the  controllers.  This  work  has  led  to  the  start  of  a  joint 
AFRL-SYTRONICS  technology  transition  effort  to  implement  speech  interface  technology  in 
operational  AWACS  aircraft  using  OSD  Challenge  and  matching  Phase  II  SBIR  funding. 


Figure  4.  Multisensory  Overview  Large-scale  Tactical  Knowledge  Environment  Laboratory 
(Major  General  Paul  Nielsen  and  General  Lester  L.  Lyles,  seated) 

Other  data  collections  required  more  subjective  and  observer  intensive  effort.  Cognitive 
Task  Analysis  performed  in  Cognitive  Systems  research  required  highly  trained  observers  to 
perform  structured  interviews  and  observations  to  identify  what  users  think  while  performing 
their  jobs  or  interacting  with  their  computer's  user  interface.  Subjective  surveys  were  usually 
collected,  even  where  automated  data  collection  was  employed,  to  gain  additional  insights  into 
performance  of  test  systems. 

Often  automated  data  collection  challenges  researchers,  due  to  the  sheer  volume  of  data 
generated.  Physiological  data  like  those  collected  in  the  Flight  Physiology  Laboratory  (FPL) 
created  such  a  problem.  SYTRONICS  used  innovative  custom  software  algorithms  to  reduce 
these  data  to  meaningful  subsets  of  manageable  size  and  content.  Additionally,  SYTRONICS 
implemented  data  storage  safeguards  by  archiving  data  to  ensure  integrity  and  prevent  loss. 
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In  these  studies,  either  a  SYTRONICS  scientist  or  subcontractor  performed  the 
appropriate  statistical  analysis  using  industry  standard  Statistical  Analysis  Software  (SAS)™. 
Our  expertise  included  all  manner  of  univariate,  bivariate,  and  multivariate  statistics,  including 
complex  analysis  of  variance  and  covariance.  The  statistical  designs  specified  in  experimental 
planning  were  executed  or  modified  to  suit  the  data  collection  so  as  not  to  bias  the  findings.  The 
results  were  formulated  into  journal  acceptable  text  statements  and  graphical  plots  for  technical 
reports  or  refereed  journals. 

Finally,  SYTRONICS  also  provided  high  level  support  and  expertise  to  help  AFRL/HEC 
researchers  accomplish  their  information  dissemination  mission.  This  included  understanding 
the  public  release  process  required  by  the  Government.  Timothy  Barry  of  SYTRONICS  served 
with  Dr.  Mark  Draper  and  Dr.  John  Reising  of  AFRL/HEC  on  a  NATO  committee  for  UAV 
interface  design.  Dr.  Bill  Marshak  recently  completed  writing  assistance  to  Don  Monk  for  a 
report  "Human  Aspects  of  Future  Command  Systems,"  integrating  notes  and  slides  from  a 
workshop  run  by  The  Technical  Cooperation  Program  which  was  a  cooperative  effort  of  the 
Australian,  Canadian,  New  Zealand,  United  Kingdom,  and  the  United  States  Governments.  Dr. 
Marshak  also  performed  a  meta-analysis  of  human  factors  errors  in  unmanned  spaceflight  that 
was  published  as  an  AFRL  technical  report.  Numerous  proceedings  articles,  technical  reports, 
and  journal  articles  have  been  produced  with  SYTRONICS'  assistance  for  the  Government  or  for 
publication  by  SYTRONICS'  researcher  staff.  These  efforts  to  disseminate  scientific 
information  included  our  co-sponsorship  of  the  12th  International  Symposium  on  Aviation 
Psychology  in  Dayton  in  April  2003.  SYTRONICS  provided  financial  support,  organized 
committee  personnel,  and  sessioned  chairs  alongside  AFRL  in  these  meetings. 

Developmental  Research 

Although  civilian  researchers  sometimes  extend  their  experimental  work  into  prototype 
development  and  testing,  these  areas  are  now  the  norm  for  military  research  work.  Rather  than 
perform  developmental  work  in  generic  settings,  military  researchers  seek  out  the  specific  user 
environment  under  the  most  realistic  conditions  possible  in  exercises,  training  flights,  and  on  rare 
occasions,  in  combat,  if  user  urgency  dictates. 

The  nature  and  scope  of  developmental  research  is  to  support  the  breadth  of  application 
areas  in  AFRL/HEC,  which  is  as  wide  as  the  nature  of  the  diverse  crew  station  applications. 
These  areas  include:  cognitive  workload  analysis  and  design,  information  fusion  and 
visualization,  synergistic  visual  and  auditory  displays  development,  crew  process/procedure/ 
performance  modeling,  intelligent  interface  agents  for  decision  support  and  data  visualization 
fusion,  automation  for  decision  support,  as  opposed  to  data  processing,  and  quantitative 
strategies  to  sustain  cognitive  performance  and  manage  fatigue. 

In  support  of  developmental  research  at  AFRL/HEC,  SYTRONICS  designed,  developed, 
fabricated,  and  installed  warfighter  interface  hardware  and  software  with  transition  to  the 
warfighter  as  a  specific  goal.  There  are  several  AFRL/HEC  systems  that  were  developed  that 
model  real-world  military  systems  and  feed  the  kill  chain.  For  example,  AFRL/HEC's  S AFIRE 
was  designed,  developed,  and  integrated  by  SYTRONICS  personnel  to  support  distributed 
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mission  interface  development  concepts  within  the  context  of  a  "system-of-systems"  to  support 
Network  Centric  Warfare  (NCW)  experimentation.  One  keystone  of  SAFIRE  is  the  new  aTCaT 
facility.  SYTRONICS  engineers  designed  the  aTCaT  Laboratory  to  facilitate  work-centered 
decision  support  for  air/space  ISR  strategy  planning  and  assessment.  This  assists  the  decision 
makers  in  visualization,  understanding,  and  creating  of  combat  decisions  to  reduce  sensor-to- 
shooter  control  path,  or  kill  chain  (see  Figures  5  and  6).  Overall,  SAFIRE  provided  effective 
coupling  of  AFRL/HEC's  simulation  assets  for  the  experimentation  of  data  fusion  algorithms  and 
decision  aids  for  individual/team  situational  awareness  under  the  auspices  of  future  military 
combat  envisioned  during  NCW. 


Multiple  and  Moving  Time  Critical  Targets  Detected  by  Multiple  ISR  Assets 
Prosecution  Using  In-Route  Strike  Assets 


Sensor  to  Shooter  Control  Path  with  ISR  Feedback  Path 
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Figure  5.  Sensor  to  Shooter  Control  Path 
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Figure  6.  Dynamic  Targeting  Cell 


SYTRONICS  was  intimately  involved  with  the  design  of  SAFIRE  since  its  inception. 
Within  the  past  five  years,  AFRL/HEC's  simulation  capabilities  have  grown  from  two  primary, 
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but  unlinked  facilities  into  an  NCW-capable  "system-of-sys terns"  that  comprises  a  distributed 
network  of  over  twenty  complex,  interconnected  pilot-in-the-loop,  and  constructive  simulation 
resources. 

Of  paramount  importance  was  SYTRONICS'  approach  to  assembling  such  a  vast  array  of 
capabilities  in  relatively  few  years,  with  only  a  few  dedicated  personnel.  Historically, 
AFRL/HEC  had  been  a  large  software  development  shop,  employing  more  than  a  dozen 
engineers  to  develop  its  simulation  capabilities  with  monolithic,  proprietary  architectures,  built 
and  maintained  in-house.  During  the  last  five  years,  SYTRONICS  demonstrated  rapid,  low-risk 
success  over  and  over  with  the  integration  of  existing  COTS  and  Government-owned  software. 
In  a  dramatic  example,  the  following  major  products/capabilities,  comprising  thousands  of  man¬ 
hours  of  development,  were  acquired  during  SYTRONICS'  tenure  and  are  now  integrated  parts 
of  SATIRE. 


Table  I 

Acquired  Products/Capabilities 


Product/Capability 

Function 

Mantis  (Quantum3-D/CG2)  ^ 

SIRE  scene  generator 

Aalchemy  (Quantum3-D) 

SIRE  PC  image  generator 

TerraVista  (Terrex) 

Visual  database  development 

PAD  (Boeing) 

40/45  Prototype  AWACS  Display 

TDF/Battlelab  (Solipsys) 

Tactical  Display  Framework 

MSCT  (Solipsys) 

Multi-Source  Correllator  Tracker 

ModSAF  (US  Army) 

Constructive  sim. 

OtbSAF 

Constructive  sim. 

JSAF  (Joint) 

Constructive  sim. 

SACS  (AFRL/HECP) 

Low  cost  desktop  A/A  fighter 

DIS 

IEEE  Std.  for  Distributed  Simulation 

CART  (AFRL/HECI) 

Time  Critical  Targeting  (TCT)  human 
performance  model 

Real  Time  Radar/ Synthetic  Aperture 
Radar  (SAR)  (Camber) 

Physics-based  radar/SAR  simulation 

Real  Time  Radar/SAR  (SAIC) 

Image-base  radar/SAR  simulation 

MIIIRO  UAV 

Multiple  UAV  control  testbed 

EAAGLES  (ASC/HP) 

Enhanced  A/A,  A/G  fighter 

VBMS  (AFRL/VACD) 

Virtual  Battle  Management  System 

SubrScene  (AFRL/VACD) 

Scene  generator 

AWACS  40/45  Sim  (Boeing,  AWACS 
SPO) 

Command/control  test  bed 

SYTRONICS’  speech  recognition  and  processing  programs  are  numerous.  For 
R&SCSIL,  we  incorporated  speech  systems  into  the  MOLTKE  AWACS,  the  Predator  Ground 
Station,  and  the  Satellite  Toolkit  for  CRISTIL.  Finally,  our  engineers  also  helped  develop  the 
Multimedia  Information  Retrieval  Interface  (MIRI)  and  took  part  in  numerous  developmental 
programs  (such  as  Arabic  Hidden  Markov  Model  Toolkit  (HTK)  model  testing,  studies  of 
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foreign-accented  speech,  and  glottal  model  testing)  for  the  SCREAM  Laboratory  to  support  Air 
Force  and  other  Government  agencies.  By  addressing  critical  design  and  integration  issues  early, 
each  of  these  programs  has  the  potential  to  be  transitioned  to  the  warfighter  in  a  very  short  period 
of  time. 


Prototype  Development 

Even  as  Air  Force  operations  expand  into  space,  the  flying  mission  remains  the  most 
demanding  and  dangerous.  Placing  equipment  in  aircraft  (and  spacecraft)— even  temporary 
modifications  for  test  and  evaluation— requires  extensive  preparation  and  perseverance.  Such 
flight  demonstrations  are  irreplaceable  for  proof-of-concept,  winning  advocacy  from  the 
warfighter,  and  for  risk  assessment  in  technology  transition.  These  demonstrations  provided 
direct  warfighter  influence  on  new  technology  development. 

Implementation  of  flight  worthy  equipment  to  enable  the  warfighter  to  develop  and 
evaluate  advanced  Concept  of  Operations  (CONOPS)  in  an  Advanced  Technology 
Demonstration/Operational  Utility  Evaluation  ( ATD/OUE )  environment  was  especially 
essential  to  the  Helmet-Mounted  Sensory  Technology/Visually-Coupled  Systems  (HMST/VCS) 
mission.  Development  of  flight  certified  equipment  in  a  timely  and  cost  effective  manner  that 
meets  the  safety  of  flight  criteria  enabled  the  warfighter  to  concentrate  on  meeting  the  ATD/OUE 
objectives. 

SYTRONICS’  general  approach  to  ensure  the  successful  development  of  flight-certified 
equipment  was  based  upon  the  proper  application  of  system  design  theory  and  practical 
experience  gained  from  previous  implementations  in  military  aircraft.  This  approach  minimized 
risk  and  enabled  our  designs  to  achieve  the  system  performance  and  operational  objectives  as 
well  as  meet  the  Electromagnetic  Interface/Electromagnetic  Compatibility  (EMI/EMC),  shock, 
vibration,  and  safety-of-flight  requirements  defined  in  the  military  specifications. 

Our  technical  staff  supported  the  ATD/OUE  conducted  by  the  422nd  Test  and  Evaluation 
Squadron  (TES)  at  Nellis  AFB  from  the  first  flight  in  1997  to  the  preparations  to  test  and 
evaluate  the  Non  Distributed  Flight  Reference  (NDFR)  Heads-Up  Display  (HUD)  application. 
This  support  included  HVI,  helmet,  Removable  Optics  Mounting  Assembly  (ROMA),  and  CRT 
test,  evaluation  and  characterization  using  the  Programmable  Image  Display  System  (PIDS)  we 
developed  to  emulate  the  aircraft  display-control-processor.  These  contributions  were  significant 
to  the  implementation  of  the  helmet  mounted  cueing  system  for  the  High  Off-Boresight  Angle 
( HOBA )  evaluation  and  CONOPS  development. 

The  SYTRONICS  HMST/VCS  design  team  was  a  major  contributor  to  the  development 
and  implementation  of  the  Integrated  Panoramic  Night  Vision  Goggle  (IPNVG)  for  the  SH21 
Program.  We  participated  at  every  level  of  the  top-down  system  design,  process  including 
Technical  Interchange  Meetings  (TIMs),  Program  Design  Reviews  (PDRs),  and  Critical 
Design  Reviews  (CDRs)  sponsored  by  Boeing,  the  prime  contractor  and  integrator.  This 
experience  provided  a  unique  opportunity  to  interact  with  the  integrator,  subcontractors,  and  the 
warfighter  to  ensure  the  system  functional  and  performance  objectives  were  met  in  the  applied 
hardware.  Our  development  of  Interface  Control  Documentation  (ICD)  relative  to  our 


21 


hardware  and  software  interfaces  minimized  risk  and  ensured  that  all  system  components 
function  together  properly. 

In  Figure  7,  we  show  the  SH21  system  block  diagram  including  the  Helmet  Interface 
Panel  (HIP)  and  Helmet  Operation  Panel  (HOP)  which  were  SYTRONICS  deliverables.  We 
also  provided  all  technical  oversight  for  the  design  and  fabrication  of  the  HVI.  Perhaps  the  most 
significant  contribution  our  team  made  was  in  the  implementation  of  the  Active  Matrix  Organic 
Light  Emitting  Diode  (AMOLED)  micro-display  image  source  into  the  IPNVG. 
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Figure  7.  SH21  System  Block  Diagram 

Successful  implementation  of  this  technology  was  ensured  by  performing  extensive 
testing  and  evaluation  to  fully  define  the  AMOLED  functional  and  performance  characteristics. 
The  migration  to  flat  panel  microdisplay  technology  represents  a  major  departure  from  the 
previous  miniature  CRT  image  source  and  enhances  helmet  application  by  reducing  weight  and 
power.  Also,  a  major  improvement  in  performance  is  achieved  by  providing  inserted  and/or 
overlaid  system/sensor  full-color  video  to  the  Night  Vision  Imaging  System  (NVIS). 
Additionally,  we  presented  the  AMOLED  image  source  for  IPNVG  work  at  the  Society  of 
Optical  Engineers  and  Scientists  (SPIE)  Defense  and  Security  Convention  in  April,  2004. 
Finally,  we  designed  and  developed  the  IPNVG  Tester,  shown  in  Figure  8,  to  characterize, 
program,  demonstrate,  validate,  and  maintain  the  IPNVG  throughout  the  ATD/OUE. 
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Figure  8.  IPNVG  Tester 


The  achievements  of  the  SYTRONICS  HMST/VCS  engineering:  team  were  well 
respected  within  the  military  HMD  community.  We  proved  we  could  work  multiple  projects 
simultaneously,  respond  quickly  to  the  ever  changing  requirements  encountered  in  a 
development  program,  and  provide  quality  HMD  and  related  systems.  Also,  SYTRONICS’  local 
presence  provided  quick-turn  technical  support  in  response  to  peak  workloads  to  accelerate 
hardware  fabrication  which  enabled  the  engineering  team  to  achieve  more  timely  Government 
Furnished  Equipment  (GFE)  deliveries. 

Installation  of  equipment  such  as  the  Line  Replaceable  Unit  ( LRU)  and  aircrew-worn 
flight  worthy  equipment  for  ATD/OUE  implementation  such  as  HIP,  HOP,  Remote  Electronic 
Chassis  (REC),  HMD,  IPNVG,  Quick  Disconnect  ( QDC '),  HVI,  and  digital  kneeboard  in  fighter 
aircraft  create  many  possible  risk  scenarios  to  the  aircrew  and  aircraft  systems.  SYTRONICS 
utilized  a  development  process  that  included  analysis,  prototyping,  and  testing  to  evaluate  all 
equipment,  as  appropriate,  relative  to  signal  integrity,  power  loading,  EMI/EMC,  ground  egress, 
ejection,  windblast,  Foreign  Object  Damage  (FOD),  shock,  vibration,  and  other  environmental 
criteria  to  mitigate  risk  and  ensure  safety  of  flight. 

Another  safety  practice  was  to  restrict  interfaces  which  interact  with  aircraft  buses  like 
the  1553  to  read-only  configurations.  Class  II  modifications  (aircraft  changes  to  incorporate 
prototype  hardware  and  software)  are  significantly  easier  if  the  systems  in  question  are  read¬ 
only.  This  turns  out  to  be  a  modest  restriction,  since  most  human  factors  augmentations  do  not 
affect  controls  directly.  If  a  system  must  exert  control  of  critical  aircraft  systems,  a  considerable 
certification  process  is  necessary  to  fly.  Waivers  can  be  received,  but  are  rare  because  of  the 
safety  concern  for  the  crew. 


Systems  Integration 


The  scope  of  system  integration  under  R&SCSIL  included  integration  of  all  manner  of 
devices,  systems,  and  modules,  including  hardware  and  software  items.  Software  module 
integration  may  have  been  at  the  executable  stage,  or  may  have  required  integration  of  source 
code.  Sometimes,  software  integration  might  have  been  as  simple  as  installing  the  software 
application,  or  as  complicated  as  implementing  specialized  interface  modules,  which  handled 
specific  interface  protocols  and  timing  issues.  Hardware  integration  required  both  physical 
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connectivity,  which  required  proper  mounting  and  alignment  (such  as  for  optical  or  mechanical 
devices),  and  electrical  connectors  (such  as  RF,  analog,  vide,  and  digital  connectors  or 
connectors  for  a  computer's  standard  (i.e.,  parallel,  serial,  and  USB)  interface  ports).  Signal 
dynamic  ranges  and  bandwidths,  and  signal  or  data  formats  were  also  considered,  and  at  times 
specialized  conversion  or  transfer  modules  were  required.  Such  integration  involved  items 
developed  within  the  laboratories,  items  developed  by  SYTRONICS  or  other  support  contractors 
or  Government  staff,  COTS /Government  off-the-shelf  (GOTS)  items,  or  items  provided  by 
companies  working  on  other  tasks  supporting  AFRL/HEC.  The  nature  of  this  activity  was  to 
achieve  the  integration  properly  and  efficiently,  in  order  to  enable  the  experimentation  to  proceed 
without  delay. 

Our  approach  took  the  highly  diverse  and  heterogeneous  situation  described  above,  and 
captured  it  within  an  integration  specification-based  methodology  to  ensure  proper  and  efficient 
systems  integration.  The  methodology  was  basically  to  prepare  an  interface  specification 
(sometimes  called  an  ICD),  to  ensure  compatibility  of  the  item  to  be  integrated  with  the  host. 
The  steps  in  this  approach  are  described  below. 

1.  Gather  information  for  host  and  item  to  be  integrated:  We  collected  host  system 
documentation,  GOTS  or  COTS  item  installation  or  operation  manuals,  prior 
integration  specifications,  and  any  other  documentation  which  we  required. 

2.  Prepare  interface  specification:  We  described,  in  quantified  and  specific  detail, 
the  physical,  electrical,  and  software  interfaces.  These  may  have  been  physical 
drawings,  electrical  schematics,  software  protocol  descriptions,  or  signal  or  data 
descriptors  (including  timing  diagrams  or  descriptions),  all  in  sufficient  detail  to 
be  useful.  For  example,  a  set  of  signal  descriptors  for  a  connector  would  include 
each  pin  number,  the  names  of  the  signals  (SIG1,  Data27,  etc.),  and  signal 
descriptions  (0  to  5V  analog,  20  to  300  GHz  RF,  0  to  255  binary,  etc.).  If 
specialized  interface  hardware  or  software  modules  were  required,  we  included 
their  specifications  in  the  interface  specification.  This  specification  also  included 
a  test  plan  for  validating  the  integration  once  it  had  been  accomplished. 

3.  Review  interface  specification:  We  typically  subjected  the  specification  to  review 
by  the  owners  of  the  item  to  be  integrated  and  the  host.  Such  review  may  have 
been  an  informal  reading  and  commenting  on  the  specification,  in  the  case  of  a 
fairly  simple  integration  task.  Conversely,  it  may  have  been  a  formal  walk¬ 
through  of  the  specifications,  with  supporting  detail  from  system  or  item  testing 
and  technical  SMEs  in  attendance,  for  complex  or  potentially  troublesome 
interface  problems. 

4.  Revise  and  reevaluate  interface:  We  incorporated  all  changes  to  the  specification 
precipitated  by  the  review  (Step  3)  then  tested  or  evaluated  the  corrections  to 
ensure  we  had  a  correct  and  accurate  interface  specification. 
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5.  Release  and  implement  the  interface  specification:  We  provided  the  final 
interface  specification  to  the  host  and  item  owners,  to  ensure  satisfactory 
integration. 

6.  Implement  the  interface:  The  host,  item  owners,  and  the  integration  task  team, 
followed  the  interface  specification:  (1)  to  ensure  application  during  item 
development,  (2)  to  design  and  implement  specialized  modules  as  needed,  (3)  to 
test  and  validate  the  host-item  integration,  and  (4)  to  achieve  satisfactory 
integration. 

This  sound,  interface-specification-based  approach  ensured  a  satisfactory  integration  of 
an  item  to  be  integrated  with  a  host  system.  We  applied  its  tenets  to  the  level  of  formality  and 
documentation  that  were  prudent,  depending  on  the  complexity  and  nature  of  each  particular 
integration  activity.  As  an  example,  although  the  Little  High-End  Airborne  Laptop  (Little 
HAL)  Program  was  suspended  prior  to  implementation  of  the  Class  II  modification,  our 
HMST/VCS  engineering  team  completed  the  integration  design,  and  development,  and  were 
prepared  to  modify  the  A 10  test  aircraft  to  integrate  the  system  components  shown  in  Figure  9. 
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Figure  9.  Little  High-End  Airborne  Laptop  (Little  HAL) 

System  integration  risk  is  substantially  reduced  by  using  the  specification-based  approach 
described  above,  offering  a  less-rigid  and  de-facto  approach  to  integration.  However,  potential 
risk  of  realizing  a  faulty  integration  is  still  possible,  as  human  error  can  permeate  even  the  most 
disciplined  specification  development.  There  are  two  ways  we  mitigated  this  risk— early 
prototyping  and  emulation. 

•  Early  prototyping  forced  the  implementation  of  the  interfaces  early  in  the 
development  of  an  item  to  be  integrated.  By  forcing  this  to  happen  early  in  the 
development,  we  got  an  early  estimation  of  interface  performance  at  the  design 
level,  and  this  often  revealed  problems  not  considered  in  the  specification 
development  stage. 
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•  Emulation  is  an  excellent  way  to  reduce  risk  for  complicated  interfaces,  and  was 
especially  useful  for  situations  such  as  integrating  externally  developed  items 
where  the  developer  and  the  host  owner  were  remotely  located.  It  was  generally 
quite  inexpensive  and  easy  to  construct  a  software  or  hardware  brassboard 
emulator  for  one  side  of  an  interface.  By  building  the  emulator  and  testing  it  with 
an  item  under  development,  we  could  test  and  measure  actual  signal  handshaking, 
levels,  and  dynamic  ranges,  and  solve  real  integration  problems  well  ahead  of 
actual  integration.  Emulation  supported  the  early  prototyping  concept,  and  was 
invaluable  in  reducing  integration  risk. 

AFRL/HEC  Sustainment:  AFRL/HEC  technical  sustainment  support  encompasses  many 
diverse  research  and  support  activities  including  facility  sustainment  and  improvement,  material 
procurement,  equipment  handling  and  tracking,  hardware  and  software  integration,  subject  pool 
management,  machining  capability,  and  computer  and  network  support.  SYTRONICS  provided 
high  quality  laboratory  sustainment  for  the  R&SCSIL  contract. 

Engineering  Improvement:  SYTRONICS  leveraged  our  superior  capabilities  to  serve 
AFRL/HEC  into  the  future  as  laboratory  areas  and  facilities  were  built  up,  redesigned,  tom 
down,  and  rebuilt  to  accommodate  the  vast  landscape  of  research  projects  required  to  enhance 
warfighter  effectiveness.  For  example,  we  re-hosted  the  SIRE  facility  converting  from  costly, 
high  maintenance,  Silicon  Graphics  machines  to  lower  cost,  easily  maintainable  PC  systems.  In 
the  SCREAM  Laboratory,  SYTRONICS  engineers  developed  a  racked  Beowulf  cluster  of  44 
processors  running  Linux  that  improved  computing  performance  for  data  crunching.  When 
combined  with  disciplined  software  improvements  to  take  advantage  of  the  paral  lel  architectures, 
huge  computing  performance  savings  were  achieved.  For  example,  a  glottal  model  algorithm 
that  took  432  hours  to  process  one  set  of  conditions,  now  only  takes  28  hours— a  15X 
improvement! 

SYTRONICS  also  provided  a  capability  for  facility  infrastructure  enhancements  by 
utilizing  in-house  and  contracted  capabilities.  Our  corporate  engineers  were  able  to  oversee  and 
complete  a  major  overhaul  of  the  B-07  area  in  developing  what  is  now  the  aTCaT  facility.  This 
included  new  flooring  and  ventilation,  ceiling  tiles  and  lighting,  wiring,  and  walls  as  well  as  new 
furniture  and  integration  of  projection  systems  and  large  plasma  displays.  The  aTCaT  facility  is 
now  a  showcase  facility  for  AFRL/HEC  for  leading  edge  technology  at  an  affordable  acquisition 
and  support  cost. 

Research  Equipment  Maintenance  and  Calibration:  SYTRONICS  operated,  maintained, 
and  calibrated  research  equipment  in  compliance  with  maintenance,  operating,  and  calibration 
procedures  utilizing  our  vast  on-site  experience.  Our  approach  was  to  establish  an  annual 
calibration  schedule  in  accordance  with  manufacturer  recommendations,  as  well  as  checking  and 
recalibrating  equipment  when  used.  For  example,  SYTRONICS  personnel  established  a 
maintenance  and  calibration  schedule  for  performing  maintenance  on  the  transmission,  the  air 
brake  system,  the  BDS5  motor  controller,  and  the  control  electronics  of  the  motorized  cart  in 
Dynamic  Visual  Assessment  Program  (DVAF). 
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SYTRONICS  personnel  are  experienced  in  dealing  with  the  procedures  and  pitfalls  of 
calibration  of  laboratory  equipment  by  PMEL.  In  AFRL/HECB,  SYTRONICS  technicians 
refined  the  process  for  calibration  of  equipment  that  must  be  completed  by  the  Original 
Equipment  Manufacturer  (OEM)  such  as  the  B  &  K  Analyzers.  This  equipment  was  stated  in 
regulation  33K-1  -1 00-1  to  be  "User  Calibration/Note  #  64  PMEL"  equipment.  However,  since 
the  OEM  would  not  supply  guidelines  for  calibration,  the  equipment  must  be  sent  away  to  the 
manufacturer.  SYTRONICS  technicians  challenged  the  Note  #  64  for  this  equipment  through 
Air  Force  Metrology  and  Calibration  (AFMETCAL)  Program  to  have  it  changed  to  Note  #  44 
and,  therefore,  had  AFMETCAL  pick  up  the  cost  of  OEM  calibration.  This  challenge  was 
successful  and  resulted  in  a  cost  saving  to  the  laboratory  of  over  $1,200  dollars  per  unit  per  year. 

By  establishing  and  following  calibration  schedules  for  equipment  not  on  the  PMEL 
account,  SYTRONICS  personnel  have  alleviated  the  problem  of  frequently  used  equipment 
being  gone  for  long  periods  of  time  at  the  manufacturer.  For  example,  Minolta  spotmeters  are 
calibrated  against  a  Hoffman  LS-658C  photometric  luminance  standard  that  is  routinely 
calibrated  at  the  manufacturer.  Calibration  factors  are  calculated,  dated,  and  attached  to  the  top 
of  each  device. 

Software  Development:  SYTRONICS  coded,  upgraded,  tested,  debugged,  calibrated, 
maintained,  and  documented  all  software  necessary  to  sustain  new  and  existing  research  programs. 
SYTRONICS  instituted  the  use  software  engineering  tools  and  current  methodologies,  such  as 
Integrated  Development  Environments  (IDEs)  and  revision  control  of  source  code  ( Concurrent 
Versions  System  (CVS)  and  SourceSafe),  at  the  beginning  of  the  R&SCSIL  contract.  In  SIRE, 
SYTRONICS  personnel  instituted  standard  software  practices  where  they  did  not  previously  exist. 
Using  a  modular,  common  core  approach,  SYTRONICS  personnel  were  able  to  share  development 
libraries  between  various  laboratories,  reducing  development  time,  thus  reducing  cost  and  risk. 
For  example,  common  core  software  is  now  in  use  among  the  Synthetic  Interface  Research  of 
UAV  Systems  (SIRUS),  SIRE,  MOLTKE,  and  Digital  Entities  for  Interface  Evaluations 
(DIGEIE)  laboratories.  In  the  aTCaT  Laboratory,  which  had  previously  been  dormant  and 
isolated  for  nearly  one  year,  the  use  of  existing  in-house  and  COTS  software  allowed 
SYTRONICS  personnel  to  bring  the  facility  online  within  the  SAFIRE  architecture  after  only  two 
weeks  of  occupancy. 

Further,  SYTRONICS  continually  promoted  the  implementation  of  effective  software 
technologies.  In  the  rapidly  changing  world  of  software  development,  it  is  especially  important  to 
evaluate  and  consider  emerging  tools  (IDEs,  compilers,  debugging  aids,  etc.),  paradigms  (objects, 
use  cases,  spiral,  etc.),  and  languages  (C++,  C#,  UML,  XML,  Java,  etc.)  as  they  emerge  and 
mature  throughout  the  industry. 

Integration  of  COTS  Hardware  and  Software:  The  use  of  COTS  hardware  and  software 
products  has  proliferated  over  the  last  five  years  widely  benefiting  the  laboratories.  Under 
R&SCSIL,  SYTRONICS  continually  integrated  new  hardware  components  and  software 
products  into  existing  laboratory  systems.  SYTRONICS  personnel  worked  closely  with  the 
Government  to  ensure  that  each  laboratories'  functional  requirements  were  met  and  new 
functional  capabilities  were  incorporated  in  order  to  support  the  need  of  each  individual  research 
program  or  experiment.  Needs  were  assessed,  and  COTS  alternatives  were  reviewed  before 
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deciding  to  embark  on  any  custom  development.  SYTRONICS'  use  of  COTS  and  GOTS 
products  reduced  risk,  decreased  time  to  experiment,  minimized  the  laboratories'  role  as  a 
"software  development  house,"  provided  large  increases  in  capabilities  in  shorter  time,  and 
promoted  collaboration  between  laboratories  with  complimentary  capabilities. 

Machining  and  Fabricating  Capability:  Under  the  R&SCSIL  contract,  our  machine  shop 
manufactured  numerous  precision-machined  components,  injection  molds,  and  electro¬ 
mechanical  assemblies.  Their  services  included:  prototyping;  engineering  assistance  in  all 
project  phases;  state-of-the-art  machining  equipment;  material  selection,  including  specialty 
alloys;  and  engineering  drawings.  Some  of  their  products  included:  a  NVG  depth-of-field 
calibration  stand,  interchangeable  NVG  close-up  lens  assemblies,  fabrication  of  a  laboratory 
crazing  meter.  Organic  Light-Emitting  (OLED)  viewing  stands,  laptop  tray  electronics  boxes, 
LCD  photometer,  infrared  terrain  boards,  Panoramic  NVG  (PNVG)  tester  unit,  EPNVG  eyepiece 
injection  mold,  and  a  3-D  holographic  imaging  gun.  SYTRONICS'  on-site  machine  shop 
received  many  accolades  for  their  quality,  timeliness,  and  service. 

Subject  Pool:  Under  R&SCSIL,  SYTRONICS  maintained  and  managed  subject  pools  for 
experimental  support.  These  took  the  form  of  "permanent"  part-time  employees,  single-use 
personnel  recruited  for  special  skills  or  conditions,  and/or  expert  pilot  or  other  SMEs.  Our 
human  research  subject  pool  manager  provided  subjects  for  all  of  the  studies  conducted  on  the 
R&SCSIL  contract,  ensuring  that  the  varied  and  often  stringent  requirements  (age,  sex, 
physiology,  hearing,  vision,  etc.)  were  met.  Our  responsibilities  included  recruiting,  screening, 
training,  scheduling,  canceling,  and  rescheduling  subjects  (as  dictated  by  the  Government  or  due 
to  equipment  failure),  escorting  subjects  on  base  (and  dealing  with  the  ever  changing 
requirements  for  that)  and  to  the  various  laboratories,  paying  subjects,  completing  financial 
reports,  and  maintaining  schedules  of  each  laboratory  as  well  as  a  record  system  to  track 
experiments,  subjects,  and  subject  payments.  We  also  visited  local  universities  and  worked  with 
their  staff  to  recruit  new  subjects. 

For  AFRL/HECB,  we  maintained  a  subject  pool  of  part-time  employees  who  were 
available  for  evaluation  of  protection  and  communication  hardware  done  on  an  ongoing  basis. 
Subject  management  and  delivery  was  particularly  difficult  after  11  September  2001,  when  the 
Air  Force  increased  security  and  restricted  access  to  the  base.  However,  despite  ever  changing 
and  confusing  security  procedures,  SYTRONICS  was  able  to  continue  supplying  subjects  during 
this  trying  time  without  disruption  to  the  ongoing  bioacoustic  research  by  the  diligence  and 
perseverance  of  our  subject  pool  manager.  Other  studies  like  the  AWACS  operator  evaluation  of 
advanced  interfaces  in  the  SIRE  facility  required  arranging  for  and  using  military  members  to 
serve  as  subjects. 

Network  Operations  Support:  SYTRONICS  provided  sustaining  effort  for  the  Warfighter 
Interface  Laboratories'  information  networks.  Under  R&SCSIL,  SYTRONICS  was  responsible 
for  the  maintenance  and  documentation  of  the  networks.  SYTRONICS  personnel  coordinated 
with  the  88th  Communications  Group  to  plan  and  document  the  laboratories'  information 
networks  to  meet  existing  and  emerging  requirements.  Furthermore,  SYTRONICS  personnel 
provided  general  network  connectivity  and  maintenance  of  PC-,  UNIX-,  and  Linux-based 
machines  and  software. 
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SYTRONICS  personnel  have  been  instrumental  in  the  development,  maintenance,  repair, 
upgrade,  enhancement,  and  integration  of  many  laboratories  onto  the  S AFIRE  simulation 
network.  SAF1RE  has  grown  during  the  last  two  years  from  one  laboratory  (SIRE)  to  the 
participation  in  distributed  simulations  of  over  eight  laboratories  (see  Figure  10).  In  support  of 
NCW  research,  the  SAFIRE  network  now  includes  SIRE,  Fusion  Immersive  for  Tactical 
Environment  (FITE),  DIGEIE  Red,  DIGEIE  Blue,  MOLTKE,  SIRUS,  aTCaT,  Simulation  and 
Analysis  Facility  (SIMAF)  (ASC),  and  the  AFRL/HECI  Centrifuge.  SYTRONICS  supported 
initial  efforts  to  integrate  SAFIRE  facilities  with  facilities  in  Building  441,  as  well  as  the  new 
Forward  Area  Controller  (FAC)  facility  in  AFRL/HECP,  and  Defense  Research  Engineering 
Network  (DREN)  connectivity  to  AFRL/HEA  at  Mesa  City  Base.  This  DREN  connectivity 
would  enable  AFRL/HEC  to  participate  in  distributed  simulation  events  such  as  Joint 
Expeditionary  Force  Experiment  (JEFX)  and  Simulation  Exercise  (SimEx). 


Simulation  Environment  Established 


C2ISR  Air-to-Air  Air-to-Ground  Constructive 


DigEIE-Blue  j 

IDtgEIE-Red 
Centrifuge 


Figure  10.  SAFIRE  Simulation  Environment 

Multimedia:  Finally,  SYTRONICS'  superb  multimedia  and  graphics  staff  supported 
AFRL/HEC  with  the  highest  level  of  graphic  design,  illustration,  animation,  3-D  modeling,  Web 
design,  video  production,  and  publication.  Our  graphics  staff  used  the  latest  in  multimedia 
software  including  the  Adobe  Creative  Suite,  Adobe  Video  Collection,  Carrara  Studio,  Amapi 
Designer,  and  Poser  3-D  Human  Modeling  to  provide  state-of-the  art  graphics  and  modeling  for 
AFRL/HEC  programs.  Our  staff  received  numerous  accolades  for  their  designs,  service,  and 
efficiency. 

Documentation:  For  all  projects,  SYTRONICS  reported  the  technical  research 

development  in  written  documents  and  conference  proceedings.  All  test  results  were 
documented  in  briefings  and  presentations,  using  graphics,  animation,  multimedia  and 
photographic  elements  as  required.  SYTRONICS  personnel  attended  conferences  and  symposia 
and  presented  the  results  of  the  developmental  research  as  required  to  transition  research  results 
to  other  Government  agencies,  industry,  and  academic  institutions.  All  research  reports  and  the 
operator's  manuals  were  submitted  to  the  Government  for  coordination. 
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PUBLICATIONS 


Many  formal  technical  reports  were  produced  and  forwarded  to  the  Defense  Technical 
Information  Center  (DTIC)  for  publication.  Selected  technical  reports  authored  or  coauthored 
by  SYTRONICS  employees  and  submitted  under  this  contract  are  summarized  in  the  ensuing 
paragraphs. 


Space  Review  Study 

Human  Factors  Engineering's  Role  in  Unmanned  Space  Operations' 

This  Space  Review  Study  was  conducted  to  ascertain  the  state-of-the-art  in  ground-based 
satellite  control,  estimate  the  impact  of  human  error  on  satellite  operations,  review  existing 
research  in  satellite  control,  and  recommend  a  human  engineering  research  and  development 
program  to  reduce  error  and  correct  deficiencies.  The  study  found  that  much  of  contemporary 
satellite  control  uses  legacy  computer  systems  and  displays  information  in  text  format,  while 
controlling  using  the  command  line  interface.  Later  systems  use  windows  and  graphical  user 
interface  technology,  which  is  only  a  small  increment  in  functionality.  The  existing  literature  on 
satellite  control  is  sparse  because  of  the  classified  nature  of  many  systems— reluctance  to  permit 
studies  and/or  data  collection,  or  because  of  a  reluctance  to  advertise  error.  The  proposed  human 
engineering  research  and  development  program  advocates  cognitive  engineering  of  displays  and 
control,  modeling  for  design,  testing  and  training,  real-time  satellite  operations,  intelligent  agents 
and/or  aiding,  monitoring  on-board  automation,  creation  of  an  operator  error  database,  and 
extending  display  and  control  guidelines.  Further,  three  different  ways  were  described  to 
facilitate  the  laboratory  space  program:  collaborative  partners,  a  common  satellite  model,  and  a 
shared  research  facility.  There  appears  to  be  sufficient  justification  for  a  human  engineering 
program  addressing  ground  control  of  space  operations.  This  research  and  development  program 
will  result  in  significant  improvement  in  operational  effectiveness,  reduced  manpower 
requirements,  and  savings  in  training  costs. 

Survey  of  Information  Visualization  Uses2 

Of  the  five  senses  humans  possess,  vision  is  by  far  the  sense  that  people  use  the  most  to 
perceive  their  environment.  With  vision,  humans  can  scan,  recognize,  and  recall  images  rapidly; 
and  can  detect  even  small  changes  in  color,  size,  shape,  movement,  or  texture.  However,  vision 
offers  more  than  just  an  amazing  array  of  reception  capability.  For  humans,  vision  is  closely 
related  to  our  thinking  processes.  Consider  the  common  sayings  "I  see!"  or  "That's  clear!"  when 
a  person  understands  something.  Perception  of  the  environment  through  vision  actually  provides 
a  large  majority  of  the  information  needed  to  perform  thinking  tasks. 

Vision  derives  its  power  through  a  combination  of  how  the  human  eye  is  constructed  and 
how  the  information  the  eye  passes  to  the  brain  is  utilized.  Basically  speaking,  the  eye  collects 
two  types  of  images:  low-resolution  for  a  large  area  and  high-resolution  for  a  small  area.  The 
brain  can  process  these  images,  in  turn,  by  either  controlled  or  attentive  processing,  or  automatic 
or  preattentive  processing.  Controlled  processing  involves  focusing  one's  gaze  at  particular 
points  in  the  field-of-view  to  get  high-resolution  images  that  are  correspondingly  information- 
rich.  Controlled  processing  is  generally  slow  and  is  characterized  by  "thinking  about"  what  one 
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sees.  On  the  other  hand,  automatic  processing  uses  the  lower-resolution  images  outside  the  focus 
of  the  controlled  processing.  Automatic  processing  is  good  at  detecting  movement  or  other 
changes  in  visual  features.  Through  automatic  processing  (analogous  to  involuntary  muscle 
movements),  the  brain  is  able  to  quickly  and  effortlessly  detect  changes  in  the  environment  that 
demand  focused  and  controlled  processing.  Another  characteristic  of  automatic  processing  is  the 
brain's  ability  to  combine  the  colors,  edges,  and  other  visual  properties  into  a  coherent 
representation  of  the  objects  being  looked  at  without  consciously  combining  the  properties  and 
recalling  from  memory;  for  example,  "That  combination  represents  a  truck."  This  behind-the- 
scenes  processing  occurs  without  thought,  occurs  continuously,  and  provides  the  large  majority 
of  the  percepts  used  by  thinking  activities.  Between  the  types  of  images  and  the  types  of 
processing,  the  vision  system  is  able  to  monitor  the  entire  field-of-view. 

Since  humans  are  visual  creatures  and  since  information  derived  from  visual  input 
provides  the  majority  of  what  feeds  our  thinking  processes,  designing  information  delivery 
systems  to  capitalize  on  these  facts  is  beneficial.  Furthermore,  other  factors  influence  improving 
information  delivery  systems,  such  as  the  sheer  amount  of  information  people  are  exposed  to  and 
must  use  daily  (providing  the  "what"  to  show)  and  advances  in  computer  graphics  and  the 
machines  that  render  them  (providing  the  "how"  to  show  it).  A  technology  area  that  combines 
these  considerations,  and  the  focus  of  this  study,  is  called  information  visualfration. 

Being  a  relatively  new  field-of-study,  the  literature  contains  several  definitions  for  what 
comprises  information  visualization.  What  information  visualization  is  not  is  the  study  of 
computer  graphics.  Although  computer  graphics  is  a  key  component  of  information 
visualization,  it  is  a  means  for  which  to  accomplish.  As  previously  discussed,  information 
visualization  is  concerned  with  affecting  thinking  processes.  The  definition  used  for  this  study 
that  captures  these  concepts— graphics  and  thinking  processes— comes  from  [Card  6]: 
"Information  Visualization:  The  use  of  computer-supported,  interactive,  visual  representations  of 
abstract  data  to  amplify  the  acquisition  or  use  of  knowledge." 

Information  Warfare  Combat  Assessment  Tool 3 

Information  Warfare  (IW)  threatens  to  overload  commanders  with  information  that 
severely  challenges  their  abilities  to  make  timely  and  effective  decisions.  Successful  IW  requires 
an  understanding  of  how  the  commander's  estimate  of  the  situation,  course-of-action  selection, 
and  subsequent  detailed  plans  are  formulated  and  executed  under  real-world  conditions.  The 
Information  Warfare  Combat  Assessment  Tool  (IWCAT)  is  an  air  campaign  decision  and 
situational  awareness  aid  for  understanding,  tracing,  and  anticipating  how  offensive  and 
defensive,  lethal  and  non-lethal,  kinetic  and  non-kinetic  actions  directed  against  adversary 
information,  information-based  processes  and  information  systems  cause  direct  tangible  and 
indirect  behavioral  effects.  This  cognitive  systems  engineering  approach  to  system  design 
produces  powerful  decision  support  system  tools  and  information  management  application 
reducing  volumes  of  complex  data  into  tailored,  cognitive  displays  to  enable  a  warfighting  staff 
and  commanders  to  rapidly  grasp  the  impact  of  current  operational  decisions  on  mission  success 
and  future  combat  capability.  While  focused  on  IW,  the  concepts  and  methodology  work  equally 
well  for  other  types  of  warfare  (e.g.,  air,  land,  sea  and/or  space  warfare)  as  well  as  across  the 
spectrum  of  conflict,  not  just  during  wartime  hostilities. 
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Speech  Interface  for  Defensive  Counterspace  Operations4 

The  primary  goal  of  Speech  Interface  for  Defensive  Counterspace  Operations  (SIDCO) 
is  to  investigate  human  interface  designs  for  the  sake  of  improving  defensive  counterspace 
operations.  SIDCO  is  being  conducted  as  an  enhancement  to  the  Center  for  Research  Support 
(CERES)  Defensive  Counterspace  Testbed  (DTB)  effort.  The  DTB  effort  aims  to  improve  the 
process  of  satellite  threat  identification  and  attack  reporting.  SIDCO  aims  to  improve  Human 
Computer  Interface  (HCI)  operations  related  to  DTB. 

Under  the  DTB  effort,  CERES  (Schriever  AFB,  CO)  is  developing  concepts  for  future 
Satellite  Operations  Squadrons  (SOPs)  to  effectively  react  to  satellite  attacks.  But  according  to 
current  policy,  satellite  problems  will  rarely  be  attributed  to  acts  of  aggression,  methods  will  be 
needed  to  clearly  distinguish  critical  events  from  normal  system  problems  or  space  weather 
incidents.  The  current  thinking  is  to  use  data  fusion  and  artificial  intelligence  to  derive  a  "best 
guess"  as  to  the  cause  and  assign  a  confidence  or  certainty  factor. 

Although  human  interface  design  was  not  the  focus  of  CERES  DTB  effort,  CERES  is 
creating  a  prototype  system  to  investigate  some  possible  displays.  For  satellite  state-of-health 
monitoring,  satellite  operations  screens  (developed  in-house  by  CERES)  are  being  used.  These 
satellite  operations  screens  are  a  major  improvement  over  the  current  screens  because  they  are 
much  more  intuitive  and  provide  better  human  computer  interaction.  When  a  significant  event 
occurs  that  causes  a  change  in  the  satellite's  health,  the  plan  is  for  a  Satellite  Toolkit  (STK)  (by 
Analytical  Graphics,  Inc.)  window  to  automatically  pop-up.  STK  improves  the  controller's 
situational  awareness  of  the  space  environment  by  providing  two-dimensional  (2-D)  and/or  3-D 
displays  of  constellations  or  individual  satellites.  The  display  will  show  the  affected  satellite  in 
3-D,  including  its  location  and  orientation  of  the  satellite  with  respect  to  Earth.  In  addition,  it 
will  also  be  possible  to  display  information  such  as  uplink  and  downlink  status  and  possibly 
unfriendly  satellites  in  the  vicinity. 

SIDCO  investigated  FICIs  for  improving  defensive  counterspace  operations. 
Specifically,  SIDCO  investigated  improving  these  operations  by  using  a  speech  recognition 
interface  into  STK  software  leveraged  on  other  yet  related  research  into  the  satellite  operator’s  C~ 
environment.  The  speech-enabled  STK  suite  provided  a  set  of  tools  representative  of  what  a 
defensive  counterspace  operator  may  use  to  perform  his  duties.  The  study  specifically  addressed 
the  question:  "Does  the  speech  recognition  interface  to  STK  improve  the  ability  to  determine 
satellite  attack?"  The  effort  resulted  in  a  demonstration  of  the  speech  enabled  STK  at  Schriever 
AFB,  CO  and  a  technical  document  reporting  the  results  of  the  study. 

Transmissivity  and  Night  Vision  Goggle  (TNVG)  Compatibility 
Data  for  Select  Aircraft  Transparencies5 

The  VIEW  facility  at  WPAFB,  OH,  has  been  evaluating  the  optical  characteristics  of 
aircraft  transparencies  for  over  25  years.  During  this  time  period,  numerous  types  of  aircraft 
transparencies  from  different  sources  have  been  measured  for  a  multitude  of  optical 
characteristics.  Some  of  these  optical  characteristics  include  angular  deviation,  multiple 


32 


imaging,  distortion,  haze,  and  transmissivity.  Though  continuous  in  nature,  transmissivity 
characteristics  can  be  divided  into  two  spectral  ranges:  visible  and  near-infrared  (NIR). 
Unaided  human  vision  is  sensitive  to  light  from  about  400  nm  (blue)  to  700  nm  (red)  with  peak 
sensitivity  occurring  at  about  555  nm  (green).  In  comparison  to  human  vision,  the  NVGs  that  are 
used  in  aircraft  cockpits  are  sensitive  to  and  amplify  light  in  the  red  and  NIR  region  of  the 
spectrum  from  about  600  nm  to  950  nm.  This  data  compilation  provided  detailed  information  on 
the  visible  and  NIR  transmissivity  characteristics  of  transparent  plastics.  These  data  allowed 
human  factors  engineers  and  vision  scientists  to  assess  the  relative  visual  impact  that  different 
types  of  transparencies  may  have  on  both  unaided  and  NVG-supplemented  aircrew  out-of-the- 
cockpit  visual  performance.  The  NIR  data  also  can  be  used  to  resolve  NVG/ transparency 
integration  issues. 

This  report  is  a  compilation  of  spectral  transmissivity  data  measure  from  numerous 
aircraft  transparencies.  The  spectral  transmissivity  of  each  part  was  measured  from  wavelengths 
of  450  nm  through  950  nm.  Some  parts  were  also  measured  at  several  different  angles  relative  to 
the  optical  axis  of  the  spectroradiometric  instrument.  The  measurements  yielded  both  visible 
light  and  NIR  spectra.  The  NIR  data  were  used  to  calculate  NVG-weighted  transmissivity 
(TNVG)  values.  TNVG  is  a  measure  of  a  transparency's  compatibility  when  it  is  used  in 
conjunction  with  NVGs.  NVGs  utilize  the  NIR  portion  (600  nm  through  950  nm)  of  the  night 
sky  ambient  illumination.  Generally  speaking,  the  higher  the  TNVG  coefficient,  the  higher  the 
NVG  visual  performance. 

Interlaboratory  Study  for  F428.83,  Standard  Test  Method  for  Intensity  of  Scratches  on 

Aerospace  Glass  Enclosures6 

The  ASTM  develops  and  publishes  standardized  test  methods.  Each  test  method  requires 
a  precision  and  bias  statement  so  organizations  that  apply  the  method  know  its  inherent 
reproducibility  (between-laboratory  variability)  and  repeatability  (within-laboratory  variability). 
Reproducibility  and  repeatability  for  this  test  method  were  determined  by  conducting  an 
Interlaboratory  Study  (ILS)  as  outlined  in  ASTM  E691 .  This  report,  which  conforms  to  the  ILS 
reporting  format  required  by  ASTM,  describes  the  study  that  was  conducted  for  ASTM  F428-83, 
Intensity  of  Scratches  on  Aerospace  Glass  Enclosures. 

Scratches  exist  on  all  glass  surfaces.  Usually,  cleaning  procedures  cause  very  fine 
scratches  that  are  not  visible  with  looking  through  the  glass.  Visible  scratches  may  be  distracting 
to  an  observer  looking  through  a  transparent  aerospace  enclosure.  Therefore,  a  procedure  to 
define  scratches  is  useful.  A  visual  comparison  was  made  between  a  set  of  graded  scratch 
standards  (adjuncts)  and  a  scratch  on  the  glass  transparency  to  determine  its  relative  intensity.  A 
vision  standard  is  used  because  it  is  not  practical  to  measure  the  dimensions  of  fine  scratches. 

ILS  for  F548-0183,  Standard  Test  Method  for  Intensity  of  Scratches  on  Aerospace 

Transparent  Plastics7 

The  ASTM  develops  and  publishes  standardized  test  methods.  Each  test  method  requires 
a  precision  and  bias  statement  so  organizations  that  apply  the  method  know  its  inherent 
reproducibility  (between-laboratory  variability)  and  repeatability  (within-laboratory  variability). 
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Reproducibility  and  repeatability  for  this  test  method  were  determined  by  conducting  an  ILS  as 
outlined  in  ASTM  E691.  This  report,  which  conforms  to  the  ILS  reporting  format  required  by 
ASTM,  describes  the  study  that  was  conducted  for  ASTM  F548-01,  Intensity  of  Scratches  on 
Aerospace  Transparent  Plastics. 

Scratches  exist  on  virtually  all  transparent  plastic  surfaces.  Usually,  cleaning  procedures 
cause  very  fine  scratches  that  are  not  visible  with  looking  through  the  plastic.  Visible  scratches 
may  be  distracting  to  an  observer  looking  through  an  aerospace  transparent  plastic.  Therefore,  a 
procedure  to  define  scratches  is  useful.  A  visual  comparison  is  made  between  a  set  of  graded 
scratch  standards  (adjuncts)  and  a  scratch  on  the  plastic  transparency  to  determine  its  relative 
intensity.  A  vision  standard  is  used  because  it  is  not  practical  to  measure  the  dimensions  of  fine 
scratches. 


Aerospace  Transparency  Research  Compendium8 

For  nearly  30  years,  the  Warfighter  Interface  Division  of  the  AFRL,  located  at  WPAFB, 
OH,  has  advanced  aerospace  transparency  technology  through  the  investigative  research  of 
numerous  optical  and  visual  parameters  inherent  in  aerospace  transparencies. 
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03  April  2001 
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Framework  (SI-0 1  -90 1 2- 1 069) 

3 1  December  200 1 

Research  and  Data— Scientific  and  Technical  Report -Air  Force  Enterprise  Defense  (AFED)  Spiral  3 
User  Manual  and  AFED  JA  VA  GUI  Source  Code  and  Application  (SI-02-90 12-1 1 1 ) 

30  January  2002 

Research  and  Data— Scientific  and  Technical  Report— Distortion  Compensated  Magnetic  Tracker  (SI- 
02-9012-265) 

10  April  2002 

Research  and  Data— Scientific  and  Technical  Report —Evaluation  of  Computer-Aided  Cognitive 

Systems  Engineering  Methodology,  Capabilities,  and  Software  (SI-02-9012-518) 

05  August  2002 

Research  and  Data— Scientific  and  Technical  Reports— Effects  of  Cognitive  Work  Analysis  (CWA)  on 
Development  of  a  Portal  for  Information  Foraging:  A  Literature  Review  (SI-02-90 1 2-575) 

23  August  2002 

Research  and  Data— Scientific  and  Technical  Reports— Final  Report  for  Teleoperation  for  Remote  Air 
Force  Space  Tasks  (SI-02-9012-575) 

23  August  2002 

Technical  Report-Human  Trust  in  Offensive  and  Defensive  Information  Operations  (SI-02-9012-617) 

1 1  September  2002 

Research  and  Data— Scientific  and  Technical  Reports— Pattern  Recognition  and  Categorization  in  the 
Liquid  Mind  System  (SI-02-9013-828) 

06  December  2002 

Research  and  Data— Scientific  and  Technical  Report—  Transmissivity  and  Night  Vision  Goggle 
Compatibility  Data  for  Select  Aircraft  Transparencies  (AFRL-HE-WP-TR-2003-00 1 5)  dated  January 
2003 

January  2003 

Research  and  Data-Scientific  and  Technical  Report-//? terlaboratory  Study  (ILS)  for  F428-83,  The 
Standard  Test  Method  for  Intensity  of  Scratches  on  Aerospace  Glass  Enclosures  (AFRL-HE-WP-TR- 
2003-00 12)  dated  January  2003 

January  2003 

Research  and  Data-Scientific  and  Technical  Report— Interlaboratory  Study  (ILS)  for  F548-0I,  The 
Standard  Test  Method for  Intensity  of  Scratches  on  Aerospace  Transparent  Plastics  (AFRL-HE-WP- 
TR-2003-0009)  dated  January  2003 

January  2003 

Research  and  Data-Scientific  and  Technical  Report -Engineering,  Software,  and  Technician  Support 
for  Simulator  Studies  to  Determine  Read-to-Control  Degradation  (SI-03-90 13-093) 

07  February  2003 
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DELIVERED 

Research  and  Data— Scientific  and  Technical  Report  —Aerospace  Transparency  Research 

Compendium  dated  June  2003  (APRL-HE-WP-TR-2003-0084) 

June  2003 

Research  and  Data— Scientific  and  Technical  Report— Seeing  Psychological  Operations  Through  A 
Cultural  Lens  (SI-03-90 1 3-428) 

1  July  2003 

Research  and  Data-Scientific  and  Technical  Reyori-Effects  of  Cognitive  Work  Analysis  (CWA)  on 
Development  of  A  Portal  for  Information  Foraging  (SI-03-9013-464) 

14  July  2003 

Research  and  Data— Scientific  and  Technical  Report— Human  Trust  in  Offensive  and  Defensive 
Information  Operations  (SI-03-9013-467) 

18  July  2003 

Research  and  Data— Scientific  and  Technical  Report— Draft  Report-Using  Activation  and  Association 
in  the  Work/Organization  Repository  of  Knowledge  (W/ORK)  Database  dated  20  October  2003 

2 1  October  2003 

Research  and  Data— Scientific  and  Technical  Report— Investigation  of  Representation  Systems  to  Aid 
Information  Collection  and  Analysis  dated  30  October  2003  (SI-03-90 14-73 1) 

30  October  2003 

Research  and  Data-Scientific  and  Technical  Report—  Windscreen  Movement  Table  Control 

Computer— Upgiude  Current  Computer  System  and  Software  dated  October  2003  (SI-03-9014-750) 

07  November  2003 

Research  and  Data-Scientific  and  Technical  Report  -Using  Activation  and  Association  in  the  W/ORK 
Database  dated  November  2003  (SI-03-90 14-801) 

20  November  2003 

Research  and  Data-Scientific  and  Technical  Report— A  Review  and  Reappraisal  of  the  Adaptive 
Human-Computer  Intel  faces  in  Complex  Control  Systems  dated  20  November  2003  (SI-03-9014-827) 

3  December  2003 

Research  and  Data-Scientific  and  Technical  Report— Investigation  of  a  Value-Based  Work-Ceniered 
Evaluation  Framework  and  Toolkit  (SI-04-90 14- 184) 

23  March  2004 

Research  and  Data-Scientific  and  Technical  Report— Cultural  Modeling  for  Command  and  Control 
(SI-04-9014-273) 

23  April  2004 

Research  and  Data-Scientific  and  (Draft)  Technical  Report —Adding  Structured  Concepts  and 
Temporal  Concept  Referents  to  the  W/ORK  Database  (SI-04-9014-523) 

28  July  2004 

Research  and  Data-Scientific  and  Technical  Report  -Effects  of  Cognitive  Work  Analysis  (CWA)  on 
Development  of  A  Portal  for  Information  Foraging  (SI-04-9014-542) 

04  August  2004 

Research  and  Data-Scientific  and  Technical  Report— IPB  Report  and  the  Knowledge  Creation 
Demonstration  from  the  A3 -IPB  Project  (Submitted  directly  to  the  Government  by  SRA) 

25  August  2004 

Research  and  Data-Scientific  and  Technical  Report— Asymmetric  Adversary  Analysis  for  Intelligent 
Preparation  of  the  Battlespace  (A3 -IPB)  and  Product  Specification  of  an  Effects-Based  Visualization 
for  Intelligence  Preparation  of  the  Battlespace  (A3 -IPB)  (SI-04-9014-634) 

30  August  2004 

Research  and  Data— Scientific  and  Technical  Report- Adding  Structured  Concepts  and  Temporal 
Concept  Referents  to  the  W/ORK  Database  (SI-04-9014-635) 

30  August  2004 

Research  and  Data-Scientific  and  Technical  Report —Development  of  Computer-Aided  Cognitive 
Systems  Engineering  Methodology,  Capabilities,  and  Software  (SI-04-90 14-667) 

13  September  2004 

Research  and  Data-Scientific  and  Technical  Report— Asymmetric  Adversary  Analysis  for  Intelligent 
Preparation  of  the  Battlespace  (A3 -IPB)— Work  Domain  Analysis  and  Design  Specification 

14  September  2004 

Research  and  Data-Scientific  and  Technical  Report— Dual  Coding  and  Self-Organization  in 
Distributed  Cognition  (DCOG)  (SI-04-9014-689) 

16  September  2004 

Research  and  Data— Scientific  and  Technical  Report —Audio/Voice  Transcription,  Timing,  and 

Analysis  Tool  (SI-04-9015-772) 

15  October  2004 

Research  and  Data-Scientific  and  Technical  Report— Enhancing  Human  Command  and  Control  for 

Air  Mission  Planning  (SI-04-90 1 5-824) 

16  November  2004 

Research  and  Data-Scientific  and  Technical  Report— Final  Report  dated  September  2005  (AFRL-HE- 
WP-TR-2005-0026) 

30  September  2005 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 


TERM 

- - - 

DEFINITION 

2-D 

Two-dimensional 

a3-ipb 

Asymmetric  Adversary  Analysis  for  Intelligent  Preparation  of  the 
Battlespace 

ADPE 

Automated  Data  Processing  Equipment 

ADT/OUE 

Advanced  Technology  Demonstration/Operational  Utility  Evaluation 

AFED 

Air  Force  Enterprise  Defense 

AFMETCAL 

Air  Force  Metrology  and  Calibration 

AFRL/HEC 

Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate, 
Warfighter  Interface  Division 

AMOLED 

Active  Matrix  Organic  Light  Emitting  Diode 

AOC 

Aerospace  Operations  Center 

ASP 

Aircrew  Speech  Processor 

ASTM 

American  Society  for  Testing  and  Materials 

aTCaT 

Augmented  Time  Critical  Analysis  and  Targeting 

AVX 

Aerospace  Vision  Program 

AWACS 

Airborne  Warning  and  Control  System 

C1 

Command  and  Control 

CDR 

Critical  Design  Review 

CERES 

Center  for  Research  Support 

COINS 

Contractor  Invoice  System 

CONOPS 

Concept  of  Operations 

COTS 

Commercial  Off-the-Shelf 

COVE 

Combat  Observation  and  Visualization  Environment 

CRDA 

Cooperative  Research  &  Development  Agreement 

CRISTIL 

Collaborative  Research  in  Space  &  Temporal  Interfaces  Laboratory 

CSRD 

Computer  System  Requirement  Document 

CVS 

Concurrent  Versions  System 

CWA 

Cognitive  Work  Analysis 

DAA 

Designated  Approval  Authority 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DCAA 

Defense  Contract  Audit  Agency 

DCOG 

Distributed  Cognition 

DFAS 

Defense  Finance  and  Accounting  Service 

DIGEIE 

Digital  Entities  for  Interface  Evaluations 

DO 

Delivery  Order 

DREN 

Defense  Research  Engineering  Network 

DTB 

Defensive  Counterspace  Testbed 

DT&E 

Display  Test  and  Evaluation 

DTIC 

Defense  Technical  Information  Center 
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TERM 

DEFINITION 

DVAF 

Dynamic  Visual  Assessment  Program 

EMI/EMC 

Electromagnetic  Interface/Electromagnetic  Compatibility 

FAC 

Forward  Area  Controller 

FITE 

Fusion  Immersive  for  Tactical  Environment 

FOD 

Foreign  Object  Damage 

FPL 

Flight  Physiology  Laboratory 

GFE 

Government  Furnished  Equipment 

GHOST 

Global  Horizontal  Operations  Support  Toolkit 

GOTS 

Government  Off-the-Shelf 

HCI 

Human  Computer  Interface 

HIP 

Helmet  Interface  Panel 

HMST 

Helmet-Mounted  Sensory  Technologies 

HMST/VCS 

Helmet-Mounted  Sensory  Technology/Visually-Coupled  Systems 

HOBA 

High  Off-Boresight  Angle 

HOP 

Helmet  Operation  Panel 

HTK 

Hidden  Markov  Model 

HUD 

Heads-Up  Display 

HVX 

Helmet  Vehicle  Interface 

ICD 

Interface  Control  Documentation 

IDE 

Integrated  Development  Environment 

ILS 

Interlaboratory  Study 

IPNVG 

Integrated  Panoramic  Night  Vision  Goggle 

IRB 

Institutional  Review  Board 

ISR 

Intelligence,  Surveillance,  and  Reconnaissance 

IW 

Information  Warfare 

IWCAT 

Information  Warfare  Combat  Assessment  Tool 

JEFX 

Joint  Expeditionary  Force  Experiment 

JTR 

Joint  Travel  Regulation 

Little  HAL 

Little  High-End  Airborne  Laptop 

LRU 

Line  Replaceable  Unit 

LTSI 

Logicon  Technical  Services,  Inc. 

MIRI 

Multimedia  Information  Retrieval  Interface 

MOCAS 

Mechanization  of  Contract  Administration  Services 

MoD 

Ministry  of  Defense 

MOLTKE 

Multisensory  Overview  Large-scale  Tactical  Knowledge  Environment 

MOU 

Memorandum  of  Understanding 

NCW 

Network  Centric  Warfare 

NDFR 

Non  Distributed  Flight  Reference 

NIR 

Near-infrared 

NVD 

Night  Vision  Device 

NVG 

Night  Vision  Goggle 

NVIS 

Night  Vision  Imaging  System 

NVO 

Night  Vision  Operations 
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TERM 

DEFINITION 

OEM 

Original  Equipment  Manufacturer 

OLED 

Organic  Light-Emitting 

OSD 

Office  of  the  Secretary  of  Defense 

OSU 

Ohio  State  University 

PA 

Project  Arrangement 

PDR 

Program  Design  Review 

pros 

Programmable  Image  Display  System 

PMEL 

Precision  Measurement  Equipment  Laboratory 

PNVG 

Panoramic  NVG 

PO 

Purchase  Order 

QDC 

Quick  Disconnect 

QPR 

Quarterly  Program  Review 

R&SCSIL 

Research  and  Sustainment  for  Crew  System  Interface  Laboratories 

RAIVE 

Research  on  Adaptive  Interfaces  for  Virtual  Environments 

RDP 

Research  and  Development  Projects 

REC 

Remote  Electronic  Chassis 

ROMA 

Removable  Optics  Mounting  Assembly 

S&E 

Scientific  and  Engineering 

SAFIRE 

Synthesized  and  Human  Aerospace  Forces  in  an  Immersion  Research 
Environment 

SAR 

Synthetic  Aperture  Radar 

SAS 

Statistical  Analysis  Software™ 

SBIR 

Small  Business  Innovation  Research 

SCREAM 

Speech  Communication  Research  Engineering  Analysis  and  Modeling 

SH21 

Strike  Helmet  21 

sroco 

Speech  Interface  for  Defensive  Counterspace  Operations 

SIMAF 

Simulation  and  Analysis  Facility 

SimEx 

Simulation  Exercise 

SIRE 

Synthesized  Immersion  Research  Environment 

SER.US 

Synthetic  Interface  Research  of  UAV  Systems 

SME 

Subject  Matter  Expert 

SOPs 

Satellite  Operations  Squadrons 

SPIE 

Society  of  Optical  Engineers  and  Scientists 

SPO 

System  Program  Office 

STE 

Synthetic  Task  Environments 

STINFO 

Scientific  and  Technical  Information 

STK 

Satellite  Toolkit 

TCT 

Time  Critical  Targeting 

TES 

Test  and  Evaluation  Squadron 

TIM 

Technical  Interchange  Meeting 

TNVG 

Transmissivity  and  Night  Vision  Goggle 

UAV 

Unmanned  Aerial  Vehicle 

UCAV 

Unmanned  Combat  Air  Vehicle 
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TERM 

DEFINITION 

UDAS 

UCAV  Decision  Aiding  System 

VBMS 

Virtual  Battle  Management  System 

VCATS 

Visually  Coupled  Acquisition  and  Targeting  System 

VDT 

Visual  Display  Technology 

VIEW 

Visual  Image  Evaluation  of  Windscreens 

WInS 

Web  Invoicing  System 

W/ORK 

Work/Organization  Repository  of  Knowledge 

WPAFB 

Wright-Patterson  Air  Force  Base 
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